ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DRY - topology construction
---------------------------
CSignalCONV and CSignalHYBRID each built the Conv+Pool front-end from scratch;
CSignalLSTM and CSignalHYBRID each built the LSTM stage from scratch. The
duplicates had already drifted: HYBRID guarded the LSTM step with
MathMax(1, historyBars/2), CSignalLSTM divided unguarded, so a historyBars of 1
gave two different steps for what is documented as the same layer.
Extracted AddConvPoolStage() and AddLstmStage() onto CExpertSignalAIBase. The
three overrides are now compositions:
CONV = AddConvPoolStage
LSTM = AddLstmStage
HYBRID = AddConvPoolStage && AddLstmStage
HYBRID's "matches the standalone CONV front-end exactly, then adds LSTM" is
enforced by construction instead of by comment. Took the guarded step for both.
Also fixed a descriptor leak the duplicates shared: on a failed topology.Add()
the CLayerDescription was neither owned by the array nor deleted.
Dead code
---------
- CNet::SaveCheckpoint / CNet::LoadCheckpoint (123 lines). Superseded by the
in-memory CaptureWeights/RestoreWeights pair; Network.mqh:1312 already said so
("This replaces the file-based SaveCheckpoint/LoadCheckpoint"). Zero call
sites - every remaining mention was a comment. The five comments that
referenced them have been reworded rather than left dangling.
- CExpertSignalCustom::CheckForDuplicateTrade / FindLastTradeIndex /
UpdateTradeStatusAndExit: declared, never defined anywhere, never called.
They only made it look as though duplicate-trade detection existed.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FileOpen(FILE_WRITE) truncates its target on open. CNet::Save already staged
the .nnw through a temp file + rename for that reason, but the three sidecars
written beside it did not:
.stats ExpertSignalAIBase.mqh:5918
.arrows ExpertSignalAIBase.mqh:6224
.cfg ExpertSignalAIBase.mqh:7329
Two defects followed.
1. An interrupted write published a truncated sidecar. For .cfg that is the
worst case: LoadAndCompareTopologyConfiguration() reads a short file as a
mismatch, which discards the trained model and restarts from era 0.
2. Windows file sharing is a mutual contract - a writer opened with no
FILE_SHARE_* blocks every concurrent open regardless of the reader's flags.
All three read paths carry FILE_SHARE_READ|FILE_SHARE_WRITE specifically so
a tester agent can read them while a live chart runs; an exclusive writer on
the same path defeated that.
Extracted CNet::Save's proven pattern into System\AtomicFile.mqh
(AtomicWriteBegin/AtomicWriteEnd) and routed all four writers through it. This
also encodes the FileMove gotcha once instead of per call site: the destination
location comes from FILE_COMMON inside the 4th arg, NOT inherited from the
source, and getting it wrong moves the file to the wrong sandbox silently.
Also fixed while in these functions:
- SaveTopologyConfiguration had 13 copy-pasted 6-line error blocks that each
returned WITHOUT FileClose(handle), leaking the handle on every write
failure. Collapsed to one ok-chain that closes exactly once. The on-disk
field order and types are unchanged (asserted during the rewrite) so existing
.cfg files still load.
- SaveChartSignals documented that pruning runs only after a successful write
("a failed write above leaves both the file AND the chart untouched") but
never checked any write result, so a partial write still deleted the chart
objects. Results are checked now, making the existing comment true.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The DFA (Direct Feedback Alignment) option was never a correct implementation:
it deterministically flipped the sign of half of all gradients based on
connection index parity, causing permanent gradient ascent for those weights
and guaranteed divergence. The backward pass was also incompatible with the
OpenCL/DirectML neuron model (layer.Total() == 1). This change removes all DFA
logic, including the enum value and `DfaFeedbackSignal` method, and replaces it
with plain gradient descent in all momentum update kernels. The `optimizer`
kernel argument is retained for binary compatibility but is no longer used.
The DFA optimizer was never real Direct Feedback Alignment — it flipped
gradient signs for half of each weight tensor deterministically, causing
permanent gradient ascent and divergence. Its backward pass was also
incompatible with the OpenCL/DirectML layer model. Remove the enum
entry, all related logic in the OpenCL kernel weight update functions,
and the `DfaFeedbackSignal` method. The `optimizer` kernel argument is
retained (unused) to preserve binary compatibility with existing DLLs.
Add type checks to correctly cast `CNeuronBaseOCL` objects in gradient clipping,
softmax evaluation, and `backProp`/`backPropDfa` loops. Previously, all neurons
were assumed to be `CNeuronBase`, causing invalid pointer casts and incorrect
gradient/output access for OpenCL layers. This ensures proper support for both
CPU and OCL neuron implementations.
Replace dynamic_cast with explicit type checks using obj.Type() and static casts in CaptureOptimizerSnapshot, RestoreOptimizerSnapshot, and SetOptimizerForAllNeurons. This avoids the runtime overhead of dynamic_cast inside critical loops, improving performance when capturing/restoring optimizer snapshots and setting optimizer for all neurons.
Changed updateInputWeights methods to accept pointer arguments by value (e.g., CLayer* instead of CLayer*&) to prevent unintended pointer reassignment and improve safety. Added overloads in CNeuronBase and CNeuronBaseOCL to support both generic CObject* and derived type pointers, unifying the virtual interface and resolving inconsistencies in network weight update logic.
Changed updateInputWeights signature from accepting `CObject*` to `CObject*&` to allow the method to modify the caller's pointer, preventing potential object copying or pointer invalidation. A large block of outdated commentary in ExpertSignalAIBase.mqh was also removed, cleaning up documentation no longer applicable after previous refactoring.
Introduce Direct Feedback Alignment (DFA) backward pass with gradient clipping, feedback matrix initialization, and a dedicated backPropDfa method. Add optimizer snapshot/restore hooks (CaptureOptimizerSnapshot, RestoreOptimizerSnapshot, SetOptimizerForAllNeurons) to temporarily switch the entire network's optimizer for replay-only updates during pass 2, preserving the original optimizer state. Support all neuron types including dropout, deconv, LSTM, and softmax in the snapshot logic.
Introduce an `optimizer` parameter to UpdateWeightsMomentum, UpdateWeightsConvMomentum, and UpdateWeightsAdam kernels. When set to a non-zero value, the gradient used for weight updates is multiplied by ±1 based on the parity of the weight index, implementing a basic feedback alignment signal for experimentation. When zero, the standard gradient is used unchanged. This allows A/B testing of alternative learning signals without modifying the rest of the training pipeline.
Cleaned up the HTML across all language versions by removing unnecessary div and br tags, restructuring headings, and fixing table formatting. Aligned the structural flow and terminology between the Chinese, Turkish, and English descriptions, corrected translation inconsistencies (e.g., mixed-language table headers in Turkish), and unified the listing of features and workflows for a more professional and consistent presentation on the marketplace.
Eliminate the separate `AI_TOPOLOGY_PRESET` enum and input.
Fold the topology presets directly into `AI_CHOICE` as new combined values (MLP_3L, MLP_4L, CONV_2L, LSTM_2L, HYBRID_2L) plus `AI_NONE`.
Remove the `TopologyPreset` input variable and update default `AIType` assignments.
Update the market description to reflect the simplified single‑selector interface.
**Why:**
Users previously had to choose an AI architecture and a topology preset separately.
Now the UI shows one coherent selector that bundles architecture with its appropriate dense‑layer depth, reducing complexity and preventing mismatches.
Allow multiple AI models to be active simultaneously by switching from
a single AIType selection to individual boolean Enable flags. Also
simplify build tag by removing date prefix.
In AI/Network.mqh, return early from InitDirectML during
tester/optimization/forward runs to prevent agent-side file-lock
failures caused by rapid stop/restart cycles accessing DLL imports.
In Expert/ExpertSignalAIBase.mqh, add MathIsValidNumber checks in
CalibratedConfidenceMagnitude and SignaledConfidence to safely handle
NaN values, and refactor ShutdownChartCleanup to accept a preserve
flag, avoiding unnecessary chart purges during tester runs for faster
shutdowns. Also add m_purgeChartOnDestruct member.
In AI/NeuronDirectML.mqh, clean up a minor comment formatting issue.
In inference-only backtests, dtStudied could be ahead of the test range, causing new-bar detection to freeze. Replaced with m_lastBarTime to keep detection aligned with runtime history. Added diagnostic logging when a non-neutral softmax output is neutralized by prior correction. Also added validation for order_price, sl, and tp in stop-checking functions to catch non-finite or negative values.
Added a new `m_modelLoadedFromDisk` flag to distinguish models loaded from a saved `.nnw` file from fresh random topologies. Modified the readiness gates in `LongCondition` and `ShortCondition` so that an inference-only tester run can execute trades using a model loaded from disk even if its persisted `trainingComplete` flag is `false`. This enables replay of a partially trained exported model without requiring full convergence, while still blocking fresh random-weight topologies from trading.
Introduce a global boolean `g_signalsVisible` to control whether signal
objects are displayed across all timeframes or hidden entirely. When
enabled, chart arrows and restore objects are set to `OBJ_ALL_PERIODS`;
otherwise they use `OBJ_NO_PERIODS`, allowing signals to be shown or
hidden at runtime without losing saved state.
Add `ShouldTraceTradeRejections()` helper that returns true only when
running in the Strategy Tester, optimization, or forward testing modes.
Use it to print diagnostic messages when trades are rejected due to a
prohibition signal or when `OpenLongParams`/`OpenShortParams` fail to
produce valid stop/take-profit levels. This provides targeted debugging
output without cluttering live trading logs.
Wrap the `TCWarnIfSlow` calls in `CExpertCustom::OnTick` and the
`TCWarnIfMemoryAbove` call in `CExpertCustom::OnTimer` with `VerboseMode`
guards. This silences routine diagnostic noise during normal operation
while still allowing detailed performance tracing when `VerboseMode`
is enabled.
Raise MIN_ACTIVATION_DERIVATIVE from 1e-4 to 1e-3 in both Network.cl and Network.mqh
to strengthen the escape signal through saturated hidden neurons. This provides a 10x
stronger safety net against fp32 OpenCL-specific saturation, complementing the earlier
output-layer fix (logit activation instead of sigmoid) that resolved the primary neutral
collapse bug.
Add MathSrand(GetTickCount()) before BuildFreshTopology() in ExpertSignalAIBase.mqh
to guarantee genuinely random weight initialization after genetic tuner evaluations,
matching the final-retrain path and Warrior_EA.mq5's OnInit. This prevents the previous
deterministic RNG state from dominating weight init.
Also remove UTF‑8 BOM from ExpertSignalAIBase.mqh and Network.mqh for cleaner encoding.
Restructured the German market description with clearer headings, an updated feature list (including structural stops and full journal), new sections for universal applicability and signals, and improved overall readability.
Add freeze-level checks, no-change modification skipping, entry price routing, and per-tick/memory budget monitoring. Override trade actions (Open, Close, Reverse, TrailingStop, TrailingOrder) to validate at the final gate before sending orders.
Introduce a custom OnTester() function to score backtest results
for genetic optimization, targeting smooth linear equity curves.
The score combines profit factor, recovery factor, Sharpe ratio,
and trade density, with a drawdown penalty.
Fixes a bug in the drawdown penalty where the percent value was
not converted to decimal before scaling, causing a near-zero
penalty. Also adjusts brace indentation for consistency.
- Append '(classic)' to RSI period 14, indicator period 14, risk-reward 1:2, and risk percent 1 preset comments
- Change Min_Vote_Close default from VOTE_CLOSE_80 to VOTE_CLOSE_DISABLED
Add MACD_FAST, MACD_SLOW, MACD_SIGNAL presets and Ichimoku Tenkan, Kijun, Senkou presets to InputEnums.mqh. All combinations are designed to satisfy the respective indicator's validation rules (fast < slow for MACD, Tenkan < Kijun < Senkou B for Ichimoku), eliminating init errors and allowing the auto-tuner to perturb settings independently.
Introduce VOTE_CLOSE_PRESETS enum with a Disabled option (value 101) that bypasses vote-driven position closing via arithmetic thresholding, removing the need for a separate boolean flag. This ensures positions exit only via stop-loss, take-profit, or trailing when disabled.
Removes standalone AI confidence parameters (MinAIConfidence, MinAIExitConfidence) and replaces them with unified Min_Vote_Open and Min_Vote_Close thresholds that apply to both AI and classic engines. Updates all code comments, report suggestions, and market descriptions accordingly, simplifying configuration and ensuring consistent vote requirements across entry and exit logic.
Add BeginVote/RevokeVote lifecycle hooks to ExpertSignalCustom and ExpertSignalAIBase.
Snapshot m_lastNonNeutralSignal before condition evaluation in Direction(), and restore
the snapshot if the vote is later discarded (e.g., Hybrid quorum shortfall).
Previously, a discarded vote still consumed the alternation gate, which could
permanently gate out valid signals until the opposite direction appeared.
:106 — window key changed from the 0-59 sec field to a full datetime. Fixing #1 alone would have replaced "always 0" with "cumulative average of every bar since startup", since the window still never rolled over.
:746 — restored result /= number. number was being counted and then never used, leaving a raw sum where the base class averages. MA's 60 + RSI's 100 = 160 tripped the ±100 range check and got zeroed — it discarded precisely the strongest agreed setups.
BlendWeightsFrom now uses CObject::Type() to correctly identify neuron classes, avoiding undefined behavior when neurons are from the plain-CPU hierarchy (CNeuron, CNeuronConv, CNeuronPool, CNeuronLSTM). Weight blending for those legacy classes is also implemented.
- Arrow deletion now scoped to rescan window (barsNow) instead of all arrows, preventing loss of multi-year arrow history on Show Signals click.
- Removed temporary throughput diagnostic counters and logging added to investigate performance regression; issue resolved.
The commit adds three counters (`m_rescanRawBuy`, `m_rescanRawSell`, `m_rescanRawNeutral`) that accumulate the raw argmax signal before the prior-correction step in `AdvanceChartSignalRescan`. This distinguishes a model that outputs mostly Neutral from one where adjusted signals suppress non-Neutral arrows. The rescan completion log now reports both the pre-decluttering tallies and the time. Additionally, `TRAIN_TIME_BUDGET_MS` is increased from 120 to 200 ms to improve throughput (as noted in the comment), balancing UI responsiveness and training speed.
The old RescanChartSignals ran the entire per-bar inference loop synchronously,
blocking the button-click handler for a potentially long duration on large lookbacks.
Replaced with StartChartSignalRescan (cheap setup) and AdvanceChartSignalRescan
(time-boxed slices) so the heavy work is drained from PollTraining's timer without
freezing the UI.
Introduce `RescanChartSignals()` method and `SIGNAL_RESCAN_LOOKBACK_BARS` define to allow operators to replace stale historical arrows (e.g., from years-old training runs) with fresh inferences from the currently deployed weights on recent bars. This prevents outdated signals from lingering on the chart and ensures the displayed set matches what a live re-render would produce.
