ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DRY - topology construction
---------------------------
CSignalCONV and CSignalHYBRID each built the Conv+Pool front-end from scratch;
CSignalLSTM and CSignalHYBRID each built the LSTM stage from scratch. The
duplicates had already drifted: HYBRID guarded the LSTM step with
MathMax(1, historyBars/2), CSignalLSTM divided unguarded, so a historyBars of 1
gave two different steps for what is documented as the same layer.
Extracted AddConvPoolStage() and AddLstmStage() onto CExpertSignalAIBase. The
three overrides are now compositions:
CONV = AddConvPoolStage
LSTM = AddLstmStage
HYBRID = AddConvPoolStage && AddLstmStage
HYBRID's "matches the standalone CONV front-end exactly, then adds LSTM" is
enforced by construction instead of by comment. Took the guarded step for both.
Also fixed a descriptor leak the duplicates shared: on a failed topology.Add()
the CLayerDescription was neither owned by the array nor deleted.
Dead code
---------
- CNet::SaveCheckpoint / CNet::LoadCheckpoint (123 lines). Superseded by the
in-memory CaptureWeights/RestoreWeights pair; Network.mqh:1312 already said so
("This replaces the file-based SaveCheckpoint/LoadCheckpoint"). Zero call
sites - every remaining mention was a comment. The five comments that
referenced them have been reworded rather than left dangling.
- CExpertSignalCustom::CheckForDuplicateTrade / FindLastTradeIndex /
UpdateTradeStatusAndExit: declared, never defined anywhere, never called.
They only made it look as though duplicate-trade detection existed.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FileOpen(FILE_WRITE) truncates its target on open. CNet::Save already staged
the .nnw through a temp file + rename for that reason, but the three sidecars
written beside it did not:
.stats ExpertSignalAIBase.mqh:5918
.arrows ExpertSignalAIBase.mqh:6224
.cfg ExpertSignalAIBase.mqh:7329
Two defects followed.
1. An interrupted write published a truncated sidecar. For .cfg that is the
worst case: LoadAndCompareTopologyConfiguration() reads a short file as a
mismatch, which discards the trained model and restarts from era 0.
2. Windows file sharing is a mutual contract - a writer opened with no
FILE_SHARE_* blocks every concurrent open regardless of the reader's flags.
All three read paths carry FILE_SHARE_READ|FILE_SHARE_WRITE specifically so
a tester agent can read them while a live chart runs; an exclusive writer on
the same path defeated that.
Extracted CNet::Save's proven pattern into System\AtomicFile.mqh
(AtomicWriteBegin/AtomicWriteEnd) and routed all four writers through it. This
also encodes the FileMove gotcha once instead of per call site: the destination
location comes from FILE_COMMON inside the 4th arg, NOT inherited from the
source, and getting it wrong moves the file to the wrong sandbox silently.
Also fixed while in these functions:
- SaveTopologyConfiguration had 13 copy-pasted 6-line error blocks that each
returned WITHOUT FileClose(handle), leaking the handle on every write
failure. Collapsed to one ok-chain that closes exactly once. The on-disk
field order and types are unchanged (asserted during the rewrite) so existing
.cfg files still load.
- SaveChartSignals documented that pruning runs only after a successful write
("a failed write above leaves both the file AND the chart untouched") but
never checked any write result, so a partial write still deleted the chart
objects. Results are checked now, making the existing comment true.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The DFA (Direct Feedback Alignment) option was never a correct implementation:
it deterministically flipped the sign of half of all gradients based on
connection index parity, causing permanent gradient ascent for those weights
and guaranteed divergence. The backward pass was also incompatible with the
OpenCL/DirectML neuron model (layer.Total() == 1). This change removes all DFA
logic, including the enum value and `DfaFeedbackSignal` method, and replaces it
with plain gradient descent in all momentum update kernels. The `optimizer`
kernel argument is retained for binary compatibility but is no longer used.
Changed updateInputWeights signature from accepting `CObject*` to `CObject*&` to allow the method to modify the caller's pointer, preventing potential object copying or pointer invalidation. A large block of outdated commentary in ExpertSignalAIBase.mqh was also removed, cleaning up documentation no longer applicable after previous refactoring.
Introduce Direct Feedback Alignment (DFA) backward pass with gradient clipping, feedback matrix initialization, and a dedicated backPropDfa method. Add optimizer snapshot/restore hooks (CaptureOptimizerSnapshot, RestoreOptimizerSnapshot, SetOptimizerForAllNeurons) to temporarily switch the entire network's optimizer for replay-only updates during pass 2, preserving the original optimizer state. Support all neuron types including dropout, deconv, LSTM, and softmax in the snapshot logic.
In AI/Network.mqh, return early from InitDirectML during
tester/optimization/forward runs to prevent agent-side file-lock
failures caused by rapid stop/restart cycles accessing DLL imports.
In Expert/ExpertSignalAIBase.mqh, add MathIsValidNumber checks in
CalibratedConfidenceMagnitude and SignaledConfidence to safely handle
NaN values, and refactor ShutdownChartCleanup to accept a preserve
flag, avoiding unnecessary chart purges during tester runs for faster
shutdowns. Also add m_purgeChartOnDestruct member.
In AI/NeuronDirectML.mqh, clean up a minor comment formatting issue.
In inference-only backtests, dtStudied could be ahead of the test range, causing new-bar detection to freeze. Replaced with m_lastBarTime to keep detection aligned with runtime history. Added diagnostic logging when a non-neutral softmax output is neutralized by prior correction. Also added validation for order_price, sl, and tp in stop-checking functions to catch non-finite or negative values.
Added a new `m_modelLoadedFromDisk` flag to distinguish models loaded from a saved `.nnw` file from fresh random topologies. Modified the readiness gates in `LongCondition` and `ShortCondition` so that an inference-only tester run can execute trades using a model loaded from disk even if its persisted `trainingComplete` flag is `false`. This enables replay of a partially trained exported model without requiring full convergence, while still blocking fresh random-weight topologies from trading.
Introduce a global boolean `g_signalsVisible` to control whether signal
objects are displayed across all timeframes or hidden entirely. When
enabled, chart arrows and restore objects are set to `OBJ_ALL_PERIODS`;
otherwise they use `OBJ_NO_PERIODS`, allowing signals to be shown or
hidden at runtime without losing saved state.
Add `ShouldTraceTradeRejections()` helper that returns true only when
running in the Strategy Tester, optimization, or forward testing modes.
Use it to print diagnostic messages when trades are rejected due to a
prohibition signal or when `OpenLongParams`/`OpenShortParams` fail to
produce valid stop/take-profit levels. This provides targeted debugging
output without cluttering live trading logs.
Raise MIN_ACTIVATION_DERIVATIVE from 1e-4 to 1e-3 in both Network.cl and Network.mqh
to strengthen the escape signal through saturated hidden neurons. This provides a 10x
stronger safety net against fp32 OpenCL-specific saturation, complementing the earlier
output-layer fix (logit activation instead of sigmoid) that resolved the primary neutral
collapse bug.
Add MathSrand(GetTickCount()) before BuildFreshTopology() in ExpertSignalAIBase.mqh
to guarantee genuinely random weight initialization after genetic tuner evaluations,
matching the final-retrain path and Warrior_EA.mq5's OnInit. This prevents the previous
deterministic RNG state from dominating weight init.
Also remove UTF‑8 BOM from ExpertSignalAIBase.mqh and Network.mqh for cleaner encoding.
Add MACD_FAST, MACD_SLOW, MACD_SIGNAL presets and Ichimoku Tenkan, Kijun, Senkou presets to InputEnums.mqh. All combinations are designed to satisfy the respective indicator's validation rules (fast < slow for MACD, Tenkan < Kijun < Senkou B for Ichimoku), eliminating init errors and allowing the auto-tuner to perturb settings independently.
Introduce VOTE_CLOSE_PRESETS enum with a Disabled option (value 101) that bypasses vote-driven position closing via arithmetic thresholding, removing the need for a separate boolean flag. This ensures positions exit only via stop-loss, take-profit, or trailing when disabled.
Removes standalone AI confidence parameters (MinAIConfidence, MinAIExitConfidence) and replaces them with unified Min_Vote_Open and Min_Vote_Close thresholds that apply to both AI and classic engines. Updates all code comments, report suggestions, and market descriptions accordingly, simplifying configuration and ensuring consistent vote requirements across entry and exit logic.
Add BeginVote/RevokeVote lifecycle hooks to ExpertSignalCustom and ExpertSignalAIBase.
Snapshot m_lastNonNeutralSignal before condition evaluation in Direction(), and restore
the snapshot if the vote is later discarded (e.g., Hybrid quorum shortfall).
Previously, a discarded vote still consumed the alternation gate, which could
permanently gate out valid signals until the opposite direction appeared.
- Arrow deletion now scoped to rescan window (barsNow) instead of all arrows, preventing loss of multi-year arrow history on Show Signals click.
- Removed temporary throughput diagnostic counters and logging added to investigate performance regression; issue resolved.
The commit adds three counters (`m_rescanRawBuy`, `m_rescanRawSell`, `m_rescanRawNeutral`) that accumulate the raw argmax signal before the prior-correction step in `AdvanceChartSignalRescan`. This distinguishes a model that outputs mostly Neutral from one where adjusted signals suppress non-Neutral arrows. The rescan completion log now reports both the pre-decluttering tallies and the time. Additionally, `TRAIN_TIME_BUDGET_MS` is increased from 120 to 200 ms to improve throughput (as noted in the comment), balancing UI responsiveness and training speed.
The old RescanChartSignals ran the entire per-bar inference loop synchronously,
blocking the button-click handler for a potentially long duration on large lookbacks.
Replaced with StartChartSignalRescan (cheap setup) and AdvanceChartSignalRescan
(time-boxed slices) so the heavy work is drained from PollTraining's timer without
freezing the UI.
Introduce `RescanChartSignals()` method and `SIGNAL_RESCAN_LOOKBACK_BARS` define to allow operators to replace stale historical arrows (e.g., from years-old training runs) with fresh inferences from the currently deployed weights on recent bars. This prevents outdated signals from lingering on the chart and ensures the displayed set matches what a live re-render would produce.
PersistWeightsOnShutdown now returns early when m_inferenceOnly is true,
preventing the tester cache from being overwritten with untrained state.
InitNeuralNetwork compares modification timestamps of the production model
(FILE_COMMON) and the agent-local cache, re-seeding when the production
copy is newer. This ensures backtests always run the latest deployed
weights even when the cache file persists across runs. Also moved
LoadChartSignals() call to after the config fingerprint is appended,
so that persisted arrows are correctly keyed per configuration.
Add FILE_SHARE_READ and FILE_SHARE_WRITE to all FileOpen calls in CNet::Load,
CNet::LoadCheckpoint, and in ExpertSignalAIBase (LoadModelStats, LoadChartSignals)
so that reading model files does not fail with error 5004 when another process
(e.g., a live chart) holds the file. Replace FileCopy with the new CopySharedFile
function for sidecar files (.cfg, .stats, _shadow.nnw) to perform share-aware
copying, preventing the same failure during tester seeding. This ensures the EA
can read and copy model artifacts even while a deployed instance is running.
Replace fixed short-delay retries in CopyFileWithRetry and LoadNetWithRetry with exponential backoff (8 attempts, cap at 2s) to reliably handle transient file locks from AV/EDR or concurrent saves.
Change autosave trigger from a fixed 300-second wall-clock timer to firing on new bar close, reducing unnecessary overwrites and narrowing the collision window with backtest FileCopy operations.
Add retry logic (CopyFileWithRetry, LoadNetWithRetry) to the tester's seeding path in ExpertSignalAIBase.mqh, and print diagnostic error messages in CNet::Load (Network.mqh) when FileOpen fails. This addresses silent failures caused by transient Windows sharing violations that occur when a live chart's atomic Save() overlaps with a Strategy Tester agent's FileCopy or FileOpen on the same production file. The new helpers retry the operation a few times with a short pause instead of silently falling back to an untrained model.
The queue duplicates minority bars up to repCount (~21x at 30.7:1 imbalance), causing IS accuracy to be scored over an ~58%-directional set while OOS counts the real ~6% distribution. This made the IS/OOS gap misleading (e.g. 77% vs 12%), appearing as catastrophic overfitting when OOS was actually stronger (1.97x vs 1.33x lift). By marking only the first occurrence per bar with `m_isTrainQueuePrimary` and using that flag in the counter, both metrics now measure the same natural class distribution, making the gap directly interpretable as generalization. Backprop and training remain unaffected—every occurrence still trains as before.
Extract shared persistence logic into PersistDeployedModel() to guarantee consistency across all deploy paths (ladder, era-cap, stop, and manual). Add DeployNow() to let the panel button finalize the current best checkpoint as the live model, and RetrainDeployed() to revert deployment and resume training from the deployed weights without starting from scratch. This enables operator-controlled deployment while preserving online continual learning.
Implement a plateau detection mechanism that escalates through warm restart, gamma annealing, and eventual deployment when balanced accuracy stagnates for `PLATEAU_PATIENCE_ERAS`. Also add a compatibility shim for the removed `MinWR` input to preserve model filenames and `.cfg` layout.
A single EA run instantiates several CNet objects (main net, EMA shadow, OOS clones, etc.). Each previously ran its own OpenCL/DirectML probe, printing redundant error banners (e.g., "OpenCL not found") and compute tier messages.
Added process-wide static flags `s_openclUnavailable` and `s_computeTierLogged` to skip subsequent probes and log once per process. This eliminates log clutter and avoids unnecessary probe overhead on hosts without OpenCL.
Rename the original `CreateElement` with a defaulted `weighScale` parameter to `CreateElementScaled` and provide a proper virtual override that matches the base `CArrayObj::CreateElement` signature exactly. This ensures `CArrayObj::Load()` dispatches correctly, fixing a bug where every saved model load failed at the first layer. Update all call sites in `CNeuronBase::Init` and `CNeuronPool::Init`. Additionally, enhance error diagnostics in `CNet::Load` to distinguish between file truncation and code faults (such as the signature mismatch).
The `Load` method now accepts an optional `quiet` flag that suppresses diagnostic
`Print` statements when set to `true`. This is used by callers that anticipate
a load failure and handle it gracefully — for example, the EMA shadow-net
bootstrap on a CPU-DLL box, which cannot hold a second full network. The main
model load keeps `quiet=false` so actual failures remain visible.
- In Network.mqh: Add upfront file size check to distinguish truncated files from backend allocation failures during model load, improving error diagnosis.
- In ExpertSignalAIBase.mqh: Remove redundant shadow net save on shutdown to avoid doubling shutdown cost and exceeding MT5's deinit budget, preventing abnormal termination and subsequent retrain.
- In Warrior_EA.mq5: Reduce training timer interval from 5s to 250ms to allow more frequent training cycles instead of sitting idle ~98% of the time.
The previous save directly wrote to the target file, which could leave a truncated/partial file if the process was force-killed (e.g., MT5 deinit timeout). Now the save writes to a temporary file (.savetmp) and only renames it over the real file after a successful write. This ensures that an interrupted save never corrupts the last good model. Additionally, improved diagnostics in CNet::Load to distinguish corrupt/incompatible files from compute errors.
Updated comments across ExpertSignalAIBase.mqh to explicitly state that the
compounded accuracy metric is a directional win-rate (Buy/Sell predictions only)
and that neutral/no-trade calls are excluded to avoid inflating the rate (since
neutral is the ~94% majority). The cumulative counters now only increment when
the prediction is directional, ensuring the panel reflects genuine trade quality
rather than overall label accuracy.
Replace the display-only MIN_FIRED_FOR_HITRATE with persistent cumulative accuracy counters (m_cumIsCorrect, m_cumIsTotal, m_cumOosCorrect, m_cumOosTotal) that survive era-to-era and restart via .stats. New ComputeCompoundedAccuracyLine() builds the panel line from these counts, providing a stable "accuracy over all validated signals so far" metric.
Split `PersistOnShutdown` into `PersistWeightsOnShutdown` (heavy weights save) and `ShutdownChartCleanup` (save arrows + purge chart). In `OnDeinit`, run panel destruction and per‑signal chart cleanup **before** the weight persistence, preventing leftover chart objects when the weight save stalls or faults past MT5's deinit budget.
Add MIN_FIRED_FOR_HITRATE constant (10) to the simple panel and training status hit-rate display. Update ScheduleTrainingIfNeeded and UpdateTrainingStatusLabel to only show a percentage when a side has at least MIN_FIRED_FOR_HITRATE live-fired calls; otherwise show "n/a" or "measuring". This prevents misleading early "0%" or "measuring" output from a single sample, while keeping full detail in logs and verbose panel.
- ResetWeights now passes `false` (not `m_isInitialized`) to SaveTopologyConfiguration so the config fingerprint matches the fresh-init write (always false). Prevents spurious "Configuration mismatch" that would discard the just-reset weights.
- UpdateTrainingStatusLabel: collapse hit-rate display to "measuring..." when neither side has fired yet; once at least one side fired, show both with "n/a" for missing side.
Add CaptureWeights and RestoreWeights methods that snapshot every neuron's weights into host arrays (CArrayDouble per neuron) and restore them in-place via setWeights. This replaces the file-based SaveCheckpoint/LoadCheckpoint for the mid-run best-era rollback, because the file path re-creates neurons (CLayer+Init) which fails on the multithreaded CPU-DLL backend (CDirectMLMy/WarriorCPU.dll) that cannot allocate a second full set of neuron tensors while the live set exists. In-memory weight copy uses only getWeights/setWeights, already proven by the per-era shadow blend. Snapshots weights only (not Adam moments); the regression handler decays eta on restore and clips per-step deltas to prevent stale-moment overshoot. Snapshot is valid only within a single Train() run. Also adds HaveWeightSnapshot() query and the m_weightSnapshot / m_haveWeightSnapshot member variables.
SaveCheckpoint and LoadCheckpoint in Network.mqh now accept a `bool common` flag. When true, the ephemeral checkpoint is stored in FILE_COMMON (co-located with the model's .nnw/.cfg/.stats/_shadow.nnw), preventing a parallel folder tree under the terminal-local Files directory. ExpertSignalAIBase.mqh is updated to pass the same flag to file existence checks and deletions, ensuring all sidecar files use the correct location.
A failed load of a corrupt/empty network could leave trainingComplete=true, causing the freshly built topology to be treated as already converged. This forced the model to never train, run inference on random weights, and delete all chart arrows. The fix ensures the state is reset to a genuine fresh start so BuildFreshTopology() actually gets trained.
- Guard CNet::Save to refuse writing a 0-layer network (prevents overwriting ~18MB model with empty stub)
- In CNet::Load and LoadCheckpoint, treat 0-layer files as load failure (older stubs still on disk)
- Introduce LOGIT_PRIOR_STRENGTH_PRESETS enum (0–100%) to control logit adjustment tau
- Prepare member variables and AdjustedSignalFromSoftmax for prior-corrected posterior at inference
- Ensure raw argmax scoring for recall/convergence remains unchanged; correction only affects live signal
Add `MA_TYPE_PRESETS` enum covering advanced (ALMA, DEMA, ZLEMA, T3, Kalman) and standard (SMA, EMA, SMMA, LWMA) moving averages. Integrate `maType` and `bestMaType` into `CADIndicatorTuner` struct, update flatten/unflatten routines, and bump `AD_TUNE_PARAM_COUNT` to 33. This allows the auto-tuner to search over MA type alongside period, improving feature discovery.
Replace single `m_pattern_0` weight with four confidence-tier weights (`m_pattern_0` through `m_pattern_3`) so that `UpdateSignalsWeights()` blends across multiple patterns like classic indicators. Previously, a single pattern caused the same win rate to be written to both the pattern weight and module weight, resulting in a quadratic derating (e.g., 70% win rate scored as 49 instead of 70). The new tiers bucket confidence into four equal bands between the minimum AI confidence and 1.0, with defaults 80/87/93/100. Added `ConfidenceTier()` and `PatternWeightForTier()` helpers, and changed default pattern count from 1 to 4.
- Added optional `weighScale` parameter (default -1.0) to `CNeuronBase::Init` and `CLayer::CreateElement`.
- Updated `CNeuronPool::Init` to use LeCun-uniform scaling (1/sqrt(window+1)) for its base initialization.
- Updated `CNet::CNet` to use He-scaled initialization (sqrt(2/neurons)) for dense layers.
- These changes enable more flexible and statistically sound weight initialization, matching the rationale used in OCL-based implementations, leading to better training stability and convergence.
