feat(ai): logit-adjusted loss, replacing oversampling and the post-hoc prior
Menon et al. 2021 (ICLR), "Long-tail learning via logit adjustment": add
tau*log(prior_c) to each class logit inside the training gradient. Softmax
CE on adjusted logits is consistent for BALANCED error - the metric
checkpoint selection already ranks on - so the loss and the deploy decision
finally optimize the same thing.
The engine already computed a true softmax + categorical-CE gradient and
wrote it over the per-neuron sigmoid delta, so this is an offset added to
three logits in the two places that gradient is built (backProp scalar path
and backPropOCL). No backend, kernel or DLL change; the forward pass and
every inference path are untouched, which is the point - the network learns
to absorb the offset, so its raw argmax becomes the balanced-optimal
decision with nothing applied at inference.
Replaces rather than stacks. Minority replay is disabled while this is on,
and the post-hoc inference prior is forced off. Stacking is not a
theoretical worry: simulated on the measured 1118/1119/34298 distribution
in the weak-signal regime, plain CE collapses to Neutral (33.4% balanced,
Buy 0%), replay reaches 48.1%, logit adjustment 50.9% with better balance -
and BOTH together score 45.4% with Neutral recall at 0%, worse than either
alone. Buda et al. 2018 predicts exactly that.
Motivation from the six-chart run: every topology took one direction to
~50% recall and abandoned the other, the direction chosen arbitrarily (the
batch-norm control went Buy 1% / Sell 42%, the inverse of the other five).
One era in 1,301 cleared the per-class recall floor.
Fingerprinted conditionally, so the converged 60.7% models on disk keep
their filenames and stay loadable as the fallback.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:05:14 -04:00
//+------------------------------------------------------------------+
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//| Warrior_EA |
//| AnimateDread |
//| |
//| Era loop, plateau ladder, checkpoint selection, deploy/finalise.|
//| |
//| PARTIAL IMPLEMENTATION FILE - not standalone. |
//| This holds CExpertSignalAIBase method BODIES only. The class |
//| declaration lives in Expert\ExpertSignalAIBase.mqh, which |
//| #includes this file at the bottom, after the declaration. Do not |
//| include it anywhere else and do not compile it on its own. |
//| |
//| Split out purely to make the 8216-line original navigable; the |
//| code inside was moved verbatim, not rewritten. |
//+------------------------------------------------------------------+
# ifndef WARRIOR_AIBASE_TRAINING_MQH
# define WARRIOR_AIBASE_TRAINING_MQH
//+------------------------------------------------------------------+
//| Training and Signal Methods |
//+------------------------------------------------------------------+
void CExpertSignalAIBase : : Train ( datetime StartTrainBar = 0 )
{
2026-08-02 01:09:18 -04:00
//--- One-shot latch so a failing forward pass reports itself ONCE per call instead of once per
//--- sample. CNet::feedForward's return value used to be discarded at all three call sites below,
//--- which is how the 2026-08-02 run spent a whole era backpropagating against a batch-norm layer
//--- whose device-side output had frozen: the only trace was 13,776 identical BufferWrite lines
//--- from three frames deeper, and nothing said training was still running on top of them.
bool forwardFailureReported = false ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
const int STABILITY_WINDOW = 3 ; // consecutive eras the OOS accuracy must hold steady for
const double STABILITY_TOLERANCE = 2.0 ; // max spread (percentage points) across that window
// Max wall-clock work per call before yielding - see m_trainRunActive's declaration comment for
// why chunking exists at all. TuneIndicatorsAndTrain()/Train() only run ONCE per dispatched
// "New Bar" custom chart event (see OnChartEventHandler - id 1001 calls it exactly once, then
// clears bEventStudy so ScheduleTrainingIfNeeded() can arm the next one), so the real throughput
// ceiling in practice is however fast MT5 itself pumps/dispatches that custom event - NOT this
// constant. Raising the OnTimer interval (5s->250ms) had ~zero effect for exactly that reason:
// ticks/chart events were already redispatching far more often than the timer alone would. Since
// per-event dispatch overhead is roughly fixed, doing more compute per event (fewer, larger
// chunks) cuts wall-clock training time roughly in proportion, but MT5 has only this one thread -
// the panel/chart can only respond to input in the gap between chunks, so 500ms made it feel
// unresponsive unless clicks landed in that narrow window. Lowered back to 120ms to keep the UI
// reactive. Raised to 200ms 2026-07-26 (throughput became the bigger complaint, as flagged above) -
// a deliberate middle ground between the reactive-but-slow 120ms and the previously-rejected 500ms,
// not a return to that. Watch panel drag/click feel after this change; back off toward 120ms if it
// regresses, or raise further only in small steps if it doesn't.
fix(ui): unique chart tag, product-grade panel, responsive under load
Three separate reports from one deploy.
1. CONV, LSTM and HYBRID all came back tagged [4109]. The weights
fingerprint omits the topology type on purpose - the file path already
separates it (State\CONV\ vs State\LSTM\ vs State\HYB\) and hashing a
value that is constant within a folder buys nothing while re-keying
every trained model into a forced retrain. So the files were never at
risk, but the tag could not do its one job. Prefixing the short id
makes it unique on the display side only; the hex half still greps
straight to the .nnw inside the folder the prefix names.
2. The default panel read like a training console. Six lines down to
three, each answering a question an owner actually has. The deploy
internals (best score, eras-since-best, ladder stage) were developer
diagnostics describing a recall floor that no longer decides anything,
and were already in the era-end journal line. In-sample accuracy left
the panel too: it grades the model on bars it trained on, so it always
flatters, and showing it beside the honest number invites reading the
wrong one. New compile-time DebuggingMode constant - deliberately not
an input - carries the IS/OOS pair and the resolved model path into
the journal instead. No extra Inputs row, no extra Market description
line, no user-reachable firehose.
3. Panel drag and buttons stuttered under training load, exactly as the
2026-07-26 note raising the chunk budget to 200ms warned they might.
Backed off to the documented 120ms - worst-case click latency is that
budget - and the derived topology (~292k weights to ~29k) makes the
throughput this costs far cheaper than when that note was written.
Also halved the panel redraw rate to 2.5 Hz: ChartRedraw repaints the
whole chart, so its cost scales with accumulated arrows, and 5 Hz was
the larger half of the stutter. Era-end still force-refreshes.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 09:05:58 -04:00
// 2026-07-30: it regressed, exactly as that warning anticipated - the panel drags stickily and
// buttons miss clicks under load, because 200ms is the worst-case latency between a click landing
// and this thread being free to notice it. Backing off to the documented 120ms. The throughput this
// costs is a far smaller sacrifice than it was when the note above was written: the derived topology
// cut the network from ~292k weights to ~29k (see ComputeFirstLayerWidth), so an era is a fraction
// of the work it used to be and the fixed per-dispatch overhead the note worried about is now a
// correspondingly smaller share of it. Responsiveness is worth more than the remainder.
const uint TRAIN_TIME_BUDGET_MS = 120 ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//---
//--- Never block the calling thread while paused/stopped - just decline this call (or finalize a
//--- run that just got stopped) and let the next scheduled call check again, so Pause/Resume/Stop
//--- and everything else on the control panel stays responsive instead of Sleep()-ing the one
//--- MQL5 thread this chart has.
if ( m_trainingPaused & & ! IsStopped ( ) & & ! m_trainingStopRequested )
return ;
bool stop = IsStopped ( ) | | m_trainingStopRequested ;
if ( stop )
{
if ( m_trainRunActive )
FinalizeTrainRun ( ) ;
if ( m_simOosRunActive )
{
delete m_simOosNet ;
m_simOosNet = NULL ;
m_simOosRunActive = false ;
}
return ;
}
//--- Evaluation-only continual-learning OOS simulation walk in progress (see StartOosContinualSimulation):
//--- give it exclusive occupancy of this call, same chunked budget as the real era loop below, so a
//--- large OOS window can't freeze the UI in one shot. While it's active no real-training
//--- ResizeBuffers()/RefreshData() runs, so the price/ATR/time buffers it reads stay frozen for its
//--- whole walk - it never has to worry about the label cache's shifting-index invalidation below.
if ( m_simOosRunActive )
{
AdvanceOosSimulationChunk ( ) ;
return ;
}
//--- Eager label-cache pre-build in progress (see StartLabelCachePrebuild/AdvanceLabelCachePrebuild) -
//--- same exclusive-occupancy/chunking treatment as the OOS simulation walk above, so it can't freeze
//--- the UI on a large study window either. m_trainRunActive stays false for its whole duration, so
//--- once it completes, Train() falls through to the normal !m_trainRunActive setup below and era 0
2026-08-01 11:27:28 -04:00
//--- starts from the measured class distribution it just seeded.
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
if ( m_labelPrebuildActive )
{
AdvanceLabelCachePrebuild ( ) ;
return ;
}
if ( ! m_trainRunActive )
{
//--- Wait (briefly, bounded, non-blocking across calls) for the terminal to finish syncing this
//--- symbol/period's history from the broker before computing the training window. Bars(symbol,
//--- period) - the hard cap on how many bars the era loop below will ever process - reflects
//--- whatever's synced SO FAR, not necessarily the true total; starting before sync completes
//--- would let that cap (and therefore the "Bar X of Y" progress display) silently grow between
//--- eras as more history trickles in.
if ( ! SeriesInfoInteger ( m_symbol . Name ( ) , PERIOD_CURRENT , SERIES_SYNCHRONIZED ) )
{
uint syncNowTick = GetTickCount ( ) ;
if ( m_syncWaitStartTick = = 0 )
m_syncWaitStartTick = syncNowTick ;
if ( syncNowTick - m_syncWaitStartTick < 5000 )
return ; // retry on the next scheduled call instead of blocking here
Print ( ID + " : WARNING - history for " + m_symbol . Name ( ) + " " + EnumToString ( PERIOD_CURRENT ) + " did not finish syncing after 5s; training window may still grow as more history arrives " ) ;
}
m_syncWaitStartTick = 0 ;
//--- 3 no-op passes before the era loop ever runs for a fresh start (see m_warmupPassesRemaining's
//--- declaration comment) - each is its own separately-scheduled Train() call (this whole method
//--- just returns, deferring to the next "New Bar"/timer-driven call), giving MT5's history sync
//--- several real, wall-clock-separated chances to settle on top of the 5s soft wait just above,
//--- before training commits to a bar count and starts populating the label cache below.
if ( m_warmupPassesRemaining > 0 )
{
m_warmupPassesRemaining - - ;
PrintVerbose ( ID + " : warm-up pass " + IntegerToString ( 3 - m_warmupPassesRemaining ) + " of 3 (letting history sync settle before training starts) " ) ;
return ;
}
feat(nn): derive dense depth, train on all history, pin the shape in .cfg
Completes the derived-topology work. Three inputs removed.
AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five
entries instead of eight. Depth is now derived from the two endpoints
the taper already has to connect (derived first-layer width, output-tied
final width) at a 2x per-layer compression target, clamped [2..5].
Asking a user to pick a layer count while the code derives the widths
those layers taper between was asking for half a decision: at 64 units
tapering to 12, four layers compress by 1.4x per step and five by 1.3x,
so the extra depth bought no abstraction. On the shipping H1/10y default
the derivation lands on 3 layers - the depth that actually won Run 2.
StudyPeriods removed. There is no case for training on less data than
the broker provides at a ~6% directional base rate; the honest
generalization read comes from the OOS holdout, not from withholding
history. Training now starts at the earliest available bar, floored by
MinTrainYear, which answers a different question (excluding dubious
pre-history) and stays.
That required closing the hazard the old code documented: the capacity
budget now MEASURES the symbol's real bar count, and a topology derived
from a measurement would widen as history downloads. Both ends are now
pinned. Every derived value left the weights-filename fingerprint -
keying a filename on a measured quantity means the EA looks for a file
that does not exist, starts from era 0 and orphans a trained model,
silently, because a missing cache is the normal first-run state. The
shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the
four derived fields rather than diffing them; a mismatch there would
discard a fully-trained model over nothing the user did. Two fields
appended to the .cfg for the conv/LSTM stages, length-guarded on read
because FileReadInteger past EOF returns 0 with no error.
ForceHiddenLayers, a compile-time constant like DebuggingMode, pins
depth for diagnostic comparisons. It joins the fingerprint only when
non-zero, so forced depths get their own files - sequential comparisons
only, not simultaneous from one .ex5.
Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64,
3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from
~58k to ~28k weights.
Both builds compile 0 errors, 0 warnings. Re-keys existing models.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
//--- ALL available history, floored by MinTrainYear. The StudyPeriods input this replaces could only
//--- ever throw data away: the signal is weak and the directional classes are rare, so every extra
//--- year is more of the minority class, and the honest generalization read comes from the OOS
//--- holdout rather than from withholding history from training. MinTrainYear survives because it
//--- answers a different question - excluding a broker's dubious pre-history - not "how much".
//--- Note the ordering: SERIES_FIRSTDATE is the floor of what EXISTS, MinTrainYear the floor of what
//--- is TRUSTED, and the window starts at whichever is later.
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
datetime firstAvailableBar = ( datetime ) SeriesInfoInteger ( m_symbol . Name ( ) , PERIOD_CURRENT , SERIES_FIRSTDATE ) ;
feat(nn): derive dense depth, train on all history, pin the shape in .cfg
Completes the derived-topology work. Three inputs removed.
AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five
entries instead of eight. Depth is now derived from the two endpoints
the taper already has to connect (derived first-layer width, output-tied
final width) at a 2x per-layer compression target, clamped [2..5].
Asking a user to pick a layer count while the code derives the widths
those layers taper between was asking for half a decision: at 64 units
tapering to 12, four layers compress by 1.4x per step and five by 1.3x,
so the extra depth bought no abstraction. On the shipping H1/10y default
the derivation lands on 3 layers - the depth that actually won Run 2.
StudyPeriods removed. There is no case for training on less data than
the broker provides at a ~6% directional base rate; the honest
generalization read comes from the OOS holdout, not from withholding
history. Training now starts at the earliest available bar, floored by
MinTrainYear, which answers a different question (excluding dubious
pre-history) and stays.
That required closing the hazard the old code documented: the capacity
budget now MEASURES the symbol's real bar count, and a topology derived
from a measurement would widen as history downloads. Both ends are now
pinned. Every derived value left the weights-filename fingerprint -
keying a filename on a measured quantity means the EA looks for a file
that does not exist, starts from era 0 and orphans a trained model,
silently, because a missing cache is the normal first-run state. The
shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the
four derived fields rather than diffing them; a mismatch there would
discard a fully-trained model over nothing the user did. Two fields
appended to the .cfg for the conv/LSTM stages, length-guarded on read
because FileReadInteger past EOF returns 0 with no error.
ForceHiddenLayers, a compile-time constant like DebuggingMode, pins
depth for diagnostic comparisons. It joins the fingerprint only when
non-zero, so forced depths get their own files - sequential comparisons
only, not simultaneous from one .ex5.
Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64,
3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from
~58k to ~28k weights.
Both builds compile 0 errors, 0 warnings. Re-keys existing models.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
MqlDateTime floor_time ;
TimeCurrent ( floor_time ) ;
floor_time . year = m_minTrainYear ;
floor_time . mon = 1 ;
floor_time . day = 1 ;
floor_time . hour = 0 ;
floor_time . min = 0 ;
floor_time . sec = 0 ;
datetime st_time = StructToTime ( floor_time ) ;
if ( firstAvailableBar > st_time )
st_time = firstAvailableBar ;
dtStudied = MathMax ( StartTrainBar , st_time ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- OOS-based objective + stability tracking: training only "converges" once the objective is
//--- met AND OOS accuracy has held inside a tight band for the last few eras, so a single lucky
//--- era can't get locked in as the final model. The best-scoring era's weights are checkpointed
//--- to an agent-local scratch file (not FILE_COMMON) and restored at the end - this works inside
//--- the tester too, unlike Net.Save()/Load() which are disabled there.
m_oosWindow . Clear ( ) ;
m_bestOosForecast = -1 ;
m_bestBalancedOos = -1 ;
m_bestPassedRecall = false ;
m_haveOosCheckpoint = false ;
m_oosStable = false ;
m_objectiveMet = false ;
m_erasSinceCooldown = 0 ;
m_eraResumePending = false ;
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- Plateau ladder starts fresh with this run, so it re-walks the escalation from its own
//--- starting point. (The focal-gamma anneal that used to reset here went with focal loss on
//--- 2026-07-31 - the ladder's real escape is the learning-rate warm restart.)
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_erasSinceBestBalanced = 0 ;
m_plateauStage = 0 ;
//--- One-time eager pre-scan for a fresh start (see m_labelCachePrebuilt's declaration comment) -
//--- kick it off and defer era 0 until it's done, so era 0 can start with a real class-balance
//--- oversampling ratio instead of the reps=1 fallback. Routed via the m_labelPrebuildActive gate
//--- above on every subsequent call until it completes.
if ( ! m_labelCachePrebuilt )
{
StartLabelCachePrebuild ( ) ;
return ;
}
m_trainRunActive = true ;
}
int bars , totalIter , oosCutoff , i ;
bool add_loop ;
if ( ! m_eraResumePending )
{
int barsNow = ( int ) MathMin ( Bars ( m_symbol . Name ( ) , PERIOD_CURRENT , dtStudied , TimeCurrent ( ) ) + m_historyBars , Bars ( m_symbol . Name ( ) , PERIOD_CURRENT ) ) ;
if ( ! ResizeBuffers ( barsNow ) | | ! RefreshData ( ) )
{
FinalizeTrainRun ( ) ;
return ;
}
bars = barsNow ;
fix(signals): revive a dead MA model, and demote Sanyaku from state to event
Two defects surfaced by research/test_classic.py, both verified fixed by re-running the
transcription against 178k bars of EURUSD H1.
CSignalMA model 1 could never fire. For any recursive average - and MA_TYPE_EMA is the
shipped default - MA(i) = a*Close(i) + (1-a)*MA(i+1), so
DiffMA(i) = a * (Close(i) - MA(i+1))
DiffCloseMA(i) = (1-a) * (Close(i) - MA(i+1))
are positive multiples of one quantity and always share a sign. Model 1 asks for a close
BELOW a RISING average, which is precisely the combination that identity forbids: 0.000%
of bars, either direction, any symbol. The MQL5 standard library this was ported from
defaults to MODE_SMA, where the two are merely correlated - the bug arrived with the EMA
default, not with the port. Reading the slope one bar back (DiffMAPrev) breaks the tie for
every MA type while keeping the model's stated meaning. Now fires on 7.92% of bars.
CSignalIchimoku model 11 fired on 27% of bars at weight 100. Sanyaku is three standing
STATES conjoined with no transition term, so it held across long stretches - and being
last in the if-chain at the top weight, the module's highest-conviction reading was also
its most common one, overwriting all eight event models below it on a quarter of all bars.
The old comment rejected an event form because "demanding all three flip on the same bar
would fire almost never" - true, but that is not the alternative. Kouten is the TURN: the
ALIGNMENT transitions, and only one role need change for it to. Testing !Sanyaku(idx+1)
fires once per aligned stretch. Now 2.17%, in line with Kumo breakout (2.4%) and the
strong TK cross (1.1%). DataReady() extended one bar deeper to cover the lookback.
Neither pattern showed edge before or after; this is about the models meaning what they
say and the vote not being dominated by a constant.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:14:34 -04:00
//--- Cross-asset panel is indexed against exactly this bar grid, so it is (re)built wherever
//--- the grid is - never per bar. Non-fatal on failure; see BuildCrossAssetPanel().
BuildCrossAssetPanel ( barsNow ) ;
feat(ai): spread as a volatility-regime feature, and fix a stale-index cache in both new blocks
Adds spread/ATR and the spread change ratio as network inputs (EnableSpreadFeature,
default on). Spread is the one microstructure channel that is both FX-available and
genuinely historical in the Strategy Tester - "during testing, the spread is not modeled
but is taken from historical data" - so unlike swap, signed tick flow or depth of market it
is something a backtest can honestly validate.
What it encodes, stated precisely because the raw measurement overstates it.
research/test_spread.py found spr/atr the strongest single feature in this codebase, on 5
of 8 instrument/geometry cells at 2-4x any volume feature. But the barrier LABEL charges
the spread inside its own barriers, so a wide-spread bar is mechanically likelier to
resolve as a loss and the feature would partly be predicting its own cost model. Relabelling
at zero cost and re-measuring the identical feature showed 20-40% of it WAS that tautology
and the majority was not (XAUUSD retained 97%). What survives is a volatility-regime
reading: spread is near-fixed while ATR is not, so the ratio runs high exactly when
realised volatility is below its own ATR estimate, which genuinely predicts whether
ATR-scaled barriers get reached. It is UNSIGNED - Neutral-vs-directional only, never a side.
Also fixes a stale-index bug I introduced with the cross-asset panel and had just repeated
in the spread series. Both cached on length alone:
if(m_crossAsset.Bars() >= bars) return true;
MQL5 series indices are relative to NOW, so one new closed candle shifts every index by
one. Keyed only on length, the panel keeps serving its index 0 as a bar that is no longer
the newest, and every cross-asset value is read one bar out of step with the price features
sitting beside it in the same vector - silently, with no error and no shape change. This is
the same class of defect as the dtStudied watermark behind the zero-direction backtests.
Both now carry a datetime anchor on m_Time.GetData(0), the same invalidation key the
label/feature bar caches already use.
And a performance fix that fell out of it: with correct invalidation the panel rebuilds on
every new bar, and RefreshConvergedSignal runs per bar - which in the tester would mean one
full multi-symbol resample per simulated bar at training depth. Inference only reads bars
0..m_historyBars-1 plus the panel's own slow window, so it now requests exactly that. The
cache check is >=, so a deeper panel left from training still satisfies it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:42:40 -04:00
EnsureSpreadSeries ( barsNow ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
add_loop = false ;
//--- Label/feature cache invalidation: MQL5 timeseries indices are always relative to "now"
//--- (index 0 = current bar), so every new closed candle shifts every older bar's index - a
//--- cache keyed by index would silently misalign the moment that happens. See
//--- EnsureBarCachesCapacity() for why `bars` + m_Time.GetData(0) are the correct/sufficient
//--- invalidation keys.
//--- When a wipe happens MID-RUN (a new candle closed while training was still going - e.g. the
//--- market reopening after the weekend), the label cache comes back empty and the lazy per-bar
//--- fallback (ComputeLabelForBar) labels everything Neutral by design (recent pivots are
//--- unconfirmable) - so continuing on a wiped cache silently turns the REST OF THE RUN into
//--- training AND scoring against an all-Neutral world. Observed 2026-07-19: eras 18-20 started
//--- right after the Sunday session open - IS error collapsed 0.44->0.22, OOS "accuracy" soared
//--- to 84.9% with Buy/Sell recall n/a and era time halved, the convergence machinery happily
//--- rewarding all-Neutral predictions on 100%-Neutral relabeled truth. Re-arm the same chunked
//--- eager prebuild that seeded era 0 and defer this era until it completes; its completion
2026-08-01 11:27:28 -04:00
//--- re-seeds the class tallies (m_prebuildSeedPending) so the next era's priors reflect the
//--- freshly relabeled window.
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
if ( EnsureBarCachesCapacity ( bars ) & & m_labelCachePrebuilt )
{
StartLabelCachePrebuild ( ) ;
return ;
}
2026-08-01 11:27:28 -04:00
//--- freeze the just-finished era's true class totals for this new era's priors (see
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- m_prevEraTrueBuyCount's declaration comment) before resetting the live counters below - EXCEPT
//--- right after StartLabelCachePrebuild()/AdvanceLabelCachePrebuild() seeded them for era 0: the
//--- live m_trueBuyCount/Sell/Neutral tally is still all-zero at that point (nothing trained yet),
2026-08-01 11:27:28 -04:00
//--- so copying it here would silently stomp the real upfront tally back to an empty distribution.
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
if ( m_prebuildSeedPending )
m_prebuildSeedPending = false ;
else
{
m_prevEraTrueBuyCount = m_trueBuyCount ;
m_prevEraTrueSellCount = m_trueSellCount ;
m_prevEraTrueNeutralCount = m_trueNeutralCount ;
}
//--- Natural class base rates for the live logit-adjusted decision (see AdjustedSignalFromSoftmax):
//--- derived from the same just-finished-era true class totals the oversampling ratio uses, so live
//--- calibrates to exactly the distribution the model was measured against. Both branches above
//--- leave m_prevEraTrue* holding the freshest real tally (prebuild-seeded on era 0, copied here
fix: the imbalance correction never ran during the auto-tune search
Neutral collapse on all four topologies by era 5 with a 2:6 barrier
(recall Buy 0% / Sell 0% / Neutral 100%), and the panel stuck on
"measuring...". One root cause, and it was not the barrier.
The labels were fine: Buy 25.4% / Sell 22.0% / Neutral 52.5%, which is
exactly gambler's ruin for m=2,k=6 (2/8 = 25% per side), with only 0.1%
of Neutral coming from the vertical barrier - so the new m*k horizon
scaling is right, arguably generous.
What was broken: Train()'s era-start block wrapped UpdateClassPriors() in
`if(!m_evalMode)`. The auto-tune GA scores every candidate in eval mode,
and AutoTuneIndicators ships ON, so on a default configuration EVERY era
of the search ran with unmeasured priors. ApplyLogitAdjustment() requires
measured priors; without them it calls ClearLogitAdjustment() and returns.
So the entire search trained under PLAIN cross-entropy. With a 52.5%
majority class the optimum of plain CE is "always predict Neutral", and
that is precisely what all four models found. The panel followed: its
counters only advance on bars the model CALLED Buy or Sell, so a
collapsed model leaves them at zero and the line reads "measuring..."
forever.
This was latent, not new. It has been true for every auto-tuned run, but
it was invisible while the labels were near-balanced - last night's
accidental 1:1 barrier gave 43/40/17, where plain CE has no majority to
collapse into. Widening the stop to 2*ATR (correctly - 1*ATR is too tight
to survive noise) moved Neutral to the majority and exposed it.
The guard's stated fear cannot happen. These priors are measured from the
LABEL distribution, and the tuner only perturbs indicator periods
(MA/RSI/MACD/Ichimoku/AD). The barrier label depends on ATR, SL_Mode and
TP_Mode - none of which the search touches - so every candidate sees
byte-identical labels and identical priors. There is nothing to
contaminate. What the guard actually protected was the .stats write, and
that is gated separately: eval candidates never checkpoint and never
persist.
Also, because this is the THIRD quiet no-op to cost a run in this
codebase (after the fictional oversampling log line and the shadow-blend
skip):
- ApplyLogitAdjustment() now WARNS when it declines to install, instead
of silently clearing. A mechanism that cannot announce it is not
running is indistinguishable from one that is.
- The panel distinguishes "measuring..." (before era 1, nothing scored
yet - an honest warm-up) from "no directional calls yet" (eras trained,
zero calls - a finding, not a wait).
Both builds compile 0 errors / 0 warnings. No retrain forced by this
commit itself, but the collapsed models must be discarded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:46:24 -04:00
//--- otherwise), so updating from them here covers both paths.
//--- 2026-08-01: THE `if(!m_evalMode)` GUARD THAT USED TO WRAP THIS IS GONE, because it silently
//--- disabled the entire imbalance correction for the whole auto-tune search. ApplyLogitAdjustment()
//--- immediately below needs measured priors; without them it clears the offsets and returns. In
//--- eval mode the priors were never measured, so every GA candidate - which is to say every era of
//--- a run with AutoTuneIndicators on, the shipped default - trained under PLAIN cross-entropy.
//--- That was invisible while the labels were near-balanced and became a total Neutral collapse the
//--- moment a 2:6 barrier put the majority class at 52.5%: recall Buy 0% / Sell 0% / Neutral 100%
//--- by era 5 on all four topologies, and the panel stuck on "measuring..." because a model that
//--- never calls a direction never accumulates a directional tally.
//--- The guard's stated fear - a search contaminating the deployed calibration - cannot happen:
//--- these priors are measured from the LABEL distribution, and the tuner only perturbs indicator
//--- periods (MA/RSI/MACD/Ichimoku/AD). The barrier label depends on ATR, SL_Mode and TP_Mode, none
//--- of which the search touches, so every candidate sees byte-identical labels and therefore
//--- identical priors. There is nothing for a candidate to contaminate. What the guard actually
//--- protected against is the .stats WRITE, and that is gated separately (eval candidates never
//--- checkpoint - see m_haveOosCheckpoint - and never persist).
UpdateClassPriors ( m_prevEraTrueBuyCount , m_prevEraTrueSellCount , m_prevEraTrueNeutralCount ) ;
feat(ai): logit-adjusted loss, replacing oversampling and the post-hoc prior
Menon et al. 2021 (ICLR), "Long-tail learning via logit adjustment": add
tau*log(prior_c) to each class logit inside the training gradient. Softmax
CE on adjusted logits is consistent for BALANCED error - the metric
checkpoint selection already ranks on - so the loss and the deploy decision
finally optimize the same thing.
The engine already computed a true softmax + categorical-CE gradient and
wrote it over the per-neuron sigmoid delta, so this is an offset added to
three logits in the two places that gradient is built (backProp scalar path
and backPropOCL). No backend, kernel or DLL change; the forward pass and
every inference path are untouched, which is the point - the network learns
to absorb the offset, so its raw argmax becomes the balanced-optimal
decision with nothing applied at inference.
Replaces rather than stacks. Minority replay is disabled while this is on,
and the post-hoc inference prior is forced off. Stacking is not a
theoretical worry: simulated on the measured 1118/1119/34298 distribution
in the weak-signal regime, plain CE collapses to Neutral (33.4% balanced,
Buy 0%), replay reaches 48.1%, logit adjustment 50.9% with better balance -
and BOTH together score 45.4% with Neutral recall at 0%, worse than either
alone. Buda et al. 2018 predicts exactly that.
Motivation from the six-chart run: every topology took one direction to
~50% recall and abandoned the other, the direction chosen arbitrarily (the
batch-norm control went Buy 1% / Sell 42%, the inverse of the other five).
One era in 1,301 cleared the per-class recall floor.
Fingerprinted conditionally, so the converged 60.7% models on disk keep
their filenames and stay loadable as the fallback.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:05:14 -04:00
//--- Re-install the training-time logit offsets from the priors just measured, so this
//--- era's gradient tracks the distribution the era is scored against. Runs in eval mode
//--- too: a GA candidate must train under the same loss as the real run or its score
//--- means nothing - only the PERSISTED calibration is withheld from eval mode.
ApplyLogitAdjustment ( ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_countBuySignals = 0 ;
m_countSellSignals = 0 ;
m_countNeutralSignals = 0 ;
m_trueBuyCount = 0 ;
m_trueSellCount = 0 ;
m_trueNeutralCount = 0 ;
m_oosBuyHits = 0 ;
m_oosBuyTotal = 0 ;
m_oosSellHits = 0 ;
m_oosSellTotal = 0 ;
m_oosNeutralHits = 0 ;
m_oosNeutralTotal = 0 ;
m_oosBuyPredicted = 0 ;
m_oosBuyPredictedHits = 0 ;
m_oosSellPredicted = 0 ;
m_oosSellPredictedHits = 0 ;
m_oosNeutralPredicted = 0 ;
m_oosNeutralPredictedHits = 0 ;
m_oosBuyFired = 0 ;
m_oosBuyFiredHits = 0 ;
m_oosSellFired = 0 ;
m_oosSellFiredHits = 0 ;
feat(ai): measure precision per confidence tier; fix stale metric labels
Two things the 2026-07-30 run exposed.
1. Every user-facing message still called the selection metric "balanced
accuracy". It has ranked on directional precision since a142749, so
"CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5%
PRECISION as if it were macro-recall, while the same era logged an
actual balanced accuracy of 49%. Two different numbers under one
name, in the line that announces a deploy. Relabelled at every site,
including the stage-3 refusal, which still described the per-class
recall floor that stopped being the gate.
2. Precision is now bucketed by confidence tier and logged per era,
both per-tier and cumulatively from each tier upward:
| tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ...
The per-tier number says whether confidence is calibrated to
correctness at all; if it does not rise T0->T3, raising the floor
buys nothing and that is the finding. The ">=" number is what a floor
would actually deliver, with its fire count, so the coverage cost is
visible in the same line. Tier weights are 25/50/75/100, so for an
AI-only config Min_Vote_Open maps straight across: 50 = ">=T1",
75 = ">=T2", 100 = ">=T3".
Bucketing happens at the existing live-fired accounting site, so it
measures exactly the population that trades - not the raw argmax.
Both builds compile 0 errors, 0 warnings. No retrain needed for either.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
ArrayInitialize ( m_oosTierFired , 0 ) ;
ArrayInitialize ( m_oosTierHits , 0 ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_oosConfidenceSum = 0 ;
// Nearest-to-present slice of this era's bars is held out as OOS and never backprop'd on;
// the rest (older bars) is the IS/training slice.
totalIter = ( int ) MathMax ( bars - MathMax ( m_historyBars , 0 ) , 0 ) ;
oosCutoff = ( int ) ( MathMax ( 0 , MathMin ( 100 , m_oosSplitPct ) ) / 100.0 * totalIter ) ;
i = ( int ) ( bars - MathMax ( m_historyBars , 0 ) - 1 ) ;
//--- Fresh era: reset pass 2's shuffled-backprop queue (see m_isTrainQueue's declaration
//--- comment). Preallocated to a parity-shaped ESTIMATE, not a hard worst case: at full parity
//--- all 3 classes replicate to ~the majority count, so the queue lands near 3x totalIter -
//--- 4x covers that plus label drift. The queueing block below grows the arrays on demand if an
//--- era ever exceeds the estimate (it used to silently DROP overflow instead - harmless at the
//--- old totalIter*cap sizing, which could never fill, but real data loss now that the measured
//--- ratio, not a small fixed cap, decides the replica count).
ArrayResize ( m_isTrainQueue , totalIter * 4 ) ;
ArrayResize ( m_isTrainQueueWeightScale , totalIter * 4 ) ;
ArrayResize ( m_isTrainQueuePrimary , totalIter * 4 ) ;
m_isTrainQueueCount = 0 ;
m_isTrainCursor = 0 ;
m_isPass2Active = false ;
m_isPass2Done = false ;
m_isPass3Active = false ;
//--- Fresh per-era predicted-signal cache for the end-of-era NMS sweep (see PruneDirectionalClusters).
//--- -2 = "not scored this era" so stale bars from a longer prior era can't draw phantom arrows.
if ( m_signalClusterWindow > 0 )
{
ArrayResize ( m_arrowSignalCache , bars ) ;
ArrayInitialize ( m_arrowSignalCache , -2.0 ) ;
}
}
else
{
//--- resuming a chunk that yielded mid-bar-loop last call - pick up exactly where it left off
bars = m_resumeBars ;
totalIter = m_resumeTotalIter ;
oosCutoff = m_resumeOosCutoff ;
add_loop = m_resumeAddLoop ;
i = m_resumeBarIndex ;
m_eraResumePending = false ;
}
// Restore this model's own learning-rate trajectory into the shared global right before this
// chunk's backProp() calls touch it - see m_modelEta's declaration comment.
eta = m_modelEta ;
uint chunkStartTick = GetTickCount ( ) ;
// Iterate over the bars - skipped entirely when resuming straight into pass 2, OR when resuming
// into a still-unfinished pass 3 (see m_isPass2Done's declaration comment for why checking
// m_isPass2Active alone isn't enough to detect the latter case): pass 1 already fully completed
// in an earlier call either way.
if ( ! m_isPass2Active & & ! m_isPass2Done )
{
for ( ; i > = 0 & & ! stop ; i - - )
{
//--- Build THIS bar's own feature window and feed it forward BEFORE checking/training against
//--- its label - see r's declaration comment below for why the window must end AT bar i, and
//--- why this must run before the label-check block rather than after: the label check needs
//--- this bar's own freshly-computed prediction, not the previous iteration's (see windowOk).
TempData . Clear ( ) ;
//--- Window ends AT (includes) bar i itself, extending m_historyBars bars into the past - i.e.
//--- "everything known as of this bar's close." Predicting label(i) - "was THIS bar the
//--- reversal" - from a window that stops short of bar i itself would blind the model to the
//--- most recent price action, which is exactly the information a reversal call most depends
//--- on. Must match RefreshLatestSignal()'s window exactly (r=i there too, i=0), since that's
//--- what actually queries the deployed model live - training on a different window than what
//--- gets queried at inference time would teach the wrong task entirely.
fix: the sequence models were reading the window backwards
BuildFeatureWindow() replaces eight hand-rolled copies of the same loop
and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first,
because MQL5 timeseries indices run backwards and `r + b` with b ascending
walks into the past.
Harmless for PAI and CONV - a dense layer learns a weight per position
either way, a conv learns time-mirrored kernels. Not harmless for the
recurrent stacks:
- LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t.
- It writes output[] only when t == steps-1: the visible output IS the
last hidden state.
- c_t = f*c_{t-1} + i*g decays toward the start of the sequence.
lstm_seq_flowcheck.cpp measured block 0's influence on the output at
1.2e-2 of block T-1's, at the shipped forget bias of 1.0.
So the bar being PREDICTED sat at the far end of the decay and the output
was handed to the OLDEST bar in the window - the exact inverse of what the
window is for. ~80x backwards on LSTM and HYBRID, on all three tiers
(OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never
surfaced as a backend discrepancy.
This does not create edge - the MI diagnostics read at the noise floor
(p=0.4975) with a working positive control. It makes the one hypothesis
those diagnostics explicitly do NOT cover testable: they are marginal and
per-bar, and state they "cannot rule out one that only exists in
combination or across time". The sequence model is the instrument for
across-time structure and it has been crippled, so that hypothesis has
never been honestly tested.
Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and
its features, so a stale .nnw would load cleanly and run a model fitted to
one ordering against the other, silently. Re-keying every config is the
point, not collateral damage. FORCES A FULL RETRAIN.
Also: the now-relative bar caches are re-keyed on the two live paths.
EnsureBarCachesCapacity() was only ever called from training paths, but
once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar
to RefreshConvergedSignal() and Train() is never re-entered - so nothing
cleared the feature cache again for the life of the process. A chart that
trained to convergence kept replaying the rows computed for the last
training era's bar grid: the live signal froze at its convergence-time
value, and OnlineLearnStep() backpropped those stale features against
freshly resolved labels. Backtests were never affected (an inference-only
process never allocates the arrays, so every read recomputes).
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 18:28:44 -04:00
//--- BuildFeatureWindow() owns the Clear/Reserve/loop AND the oldest-bar-first ordering that
//--- the LSTM stacks depend on - see its definition comment.
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
int r = i ;
bool windowOk = false ;
double displayNeuron0 = 0 , displayNeuron1 = 0 , displayNeuron2 = 0 ;
if ( r < = bars )
{
fix: the sequence models were reading the window backwards
BuildFeatureWindow() replaces eight hand-rolled copies of the same loop
and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first,
because MQL5 timeseries indices run backwards and `r + b` with b ascending
walks into the past.
Harmless for PAI and CONV - a dense layer learns a weight per position
either way, a conv learns time-mirrored kernels. Not harmless for the
recurrent stacks:
- LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t.
- It writes output[] only when t == steps-1: the visible output IS the
last hidden state.
- c_t = f*c_{t-1} + i*g decays toward the start of the sequence.
lstm_seq_flowcheck.cpp measured block 0's influence on the output at
1.2e-2 of block T-1's, at the shipped forget bias of 1.0.
So the bar being PREDICTED sat at the far end of the decay and the output
was handed to the OLDEST bar in the window - the exact inverse of what the
window is for. ~80x backwards on LSTM and HYBRID, on all three tiers
(OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never
surfaced as a backend discrepancy.
This does not create edge - the MI diagnostics read at the noise floor
(p=0.4975) with a working positive control. It makes the one hypothesis
those diagnostics explicitly do NOT cover testable: they are marginal and
per-bar, and state they "cannot rule out one that only exists in
combination or across time". The sequence model is the instrument for
across-time structure and it has been crippled, so that hypothesis has
never been honestly tested.
Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and
its features, so a stale .nnw would load cleanly and run a model fitted to
one ordering against the other, silently. Re-keying every config is the
point, not collateral damage. FORCES A FULL RETRAIN.
Also: the now-relative bar caches are re-keyed on the two live paths.
EnsureBarCachesCapacity() was only ever called from training paths, but
once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar
to RefreshConvergedSignal() and Train() is never re-entered - so nothing
cleared the feature cache again for the life of the process. A chart that
trained to convergence kept replaying the rows computed for the last
training era's bar grid: the live signal froze at its convergence-time
value, and OnlineLearnStep() backpropped those stale features against
freshly resolved labels. Backtests were never affected (an inference-only
process never allocates the arrays, so every read recomputes).
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 18:28:44 -04:00
windowOk = BuildFeatureWindow ( r ) ;
if ( windowOk )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
add_loop = true ;
}
//--- Determine label/queue-eligibility BEFORE running any feedForward this bar - see
//--- wouldQueue's use below for why. Mirrors the label-check condition this block used to
//--- gate on (moved earlier, unchanged).
bool haveLabel = false , buy = false , sell = false , wouldQueue = false ;
if ( windowOk & & i < ( int ) ( bars - MathMax ( m_historyBars , 0 ) - 1 ) & & i > 1 & & m_Time . GetData ( i ) > dtStudied
& & ( m_outputNeuronsCount = = 1 | | m_outputNeuronsCount = = 3 ) )
{
//--- The fractal/swing-confirmation/trend-context label at now-relative index i only depends
//--- on price/ATR history, never on model state, so it's identical every era until a new bar
//--- closes and shifts the index frame (see the cache invalidation check above) - cache it
//--- rather than recomputing from scratch every single era. Usually already populated by
//--- AdvanceLabelCachePrebuild() before era 0 ever starts - this is just a lazy fallback for
//--- any index it didn't cover (e.g. bars/window drifted between prebuild and era 0's start).
if ( m_labelCacheHasValue [ i ] )
{
buy = m_labelCacheBuy [ i ] ;
sell = m_labelCacheSell [ i ] ;
}
else
{
ComputeLabelForBar ( i , bars , buy , sell ) ;
m_labelCacheBuy [ i ] = buy ;
m_labelCacheSell [ i ] = sell ;
m_labelCacheHasValue [ i ] = true ;
}
haveLabel = true ;
bool isOOS = ( i < oosCutoff ) ;
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
// Embargo: a bar's triple-barrier label is decided by the m_barrierHorizonBars bars that
// follow it (see TripleBarrierLabel()). An IS bar within that distance of the OOS boundary
// therefore carries a label that was only knowable using price action from inside the
// held-out OOS window - purge that narrow band from backprop entirely instead of training
// on it as ordinary IS. Lopez de Prado ch. 7 calls this purging, and it is the whole reason
// a naive train/test split leaks on overlapping-horizon financial labels.
// Was m_swingConfirmationBars + LABEL_WINDOW_BARS, which measured the ZigZag repainting
// delay - the correct quantity for the old target and the wrong one for this label.
int embargoBars = MathMax ( m_barrierHorizonBars , 1 ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
bool isEmbargoed = ( ! isOOS & & i < oosCutoff + embargoBars ) ;
wouldQueue = ( ! isOOS & & ! isEmbargoed ) ;
}
//--- Only run this bar's feedForward (and the display/count/chart-draw work that depends on
//--- it) when pass 2 ISN'T about to redo it anyway. A queued bar gets a completely fresh
//--- feedForward moments later in pass 2 (see m_isTrainQueue's declaration comment - other
//--- queued bars ahead of it in the shuffled order may already have updated weights, so pass
//--- 2 can't reuse this scan's result even if it wanted to) - running it here too was one
//--- full forward pass per training sample thrown straight in the trash every era, on top of
//--- the one pass 2 actually needs. The book's SGD (references\neuronetworksbook.pdf, section
//--- 1.4) is one forward+backward pass per training sample, not two.
2026-08-02 01:09:18 -04:00
//--- Display/IS-scoring only on this path, but the same rule applies: getResults() after a
//--- failed pass returns the previous bar's activations, which would be shown on the panel and
//--- counted as this bar's prediction.
if ( windowOk & & ! wouldQueue & & Net . feedForward ( TempData ) )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
Net . getResults ( TempData ) ;
if ( m_outputNeuronsCount = = 1 )
dPrevSignal = TempData [ 0 ] ;
else
if ( m_outputNeuronsCount = = 3 )
dPrevSignal = ApplyClassificationSoftmax ( ) ;
//--- Snapshot the just-computed neuron output(s) for the status label display below, before
//--- the label-check block clears/refills TempData with the target label (Step A always
//--- runs after this point now) - reading TempData directly for display after that would
//--- show the TRUE LABEL of the bar just trained on, not the network's own prediction.
if ( TempData . Total ( ) > 0 )
displayNeuron0 = TempData [ 0 ] ;
if ( TempData . Total ( ) > 1 )
displayNeuron1 = TempData [ 1 ] ;
if ( TempData . Total ( ) > 2 )
displayNeuron2 = TempData [ 2 ] ;
switch ( DoubleToSignal ( dPrevSignal ) )
{
case Buy :
m_countBuySignals + + ;
break ;
case Sell :
m_countSellSignals + + ;
break ;
default :
m_countNeutralSignals + + ;
break ;
}
m_lastBarTime = m_Time . GetData ( i ) ;
if ( i > 0 )
{
// NMS on: record only - the era-end sweep is the SOLE renderer, so no raw (un-
// declustered) arrow is ever drawn mid-era. NMS off: draw inline as before.
if ( m_signalClusterWindow > 0 )
{
if ( i < ArraySize ( m_arrowSignalCache ) )
m_arrowSignalCache [ i ] = dPrevSignal ;
}
else
if ( DoubleToSignal ( dPrevSignal ) = = Neutral )
DeleteObject ( m_lastBarTime ) ;
else
DrawObject ( m_lastBarTime , dPrevSignal , m_High . GetData ( i ) , m_Low . GetData ( i ) ) ;
}
UpdateTrainingStatusLabel (
StringFormat ( " Bar %d of %d -> %.2f%% (scan) " , bars - i + 1 , bars , ( double ) ( bars - i + 1.0 ) / bars * 100 ) ,
displayNeuron0 , displayNeuron1 , displayNeuron2 , dPrevSignal ) ;
}
if ( haveLabel )
{
// True label as an ENUM_SIGNAL, derived directly from the buy/sell bools - not read
// back from TempData, which no longer holds a target at this point at all (see above).
ENUM_SIGNAL trueSignal = buy ? Buy : ( sell ? Sell : Neutral ) ;
// Track the true class distribution this era (used below to weight IS oversampling,
// and surfaced in the status label text alongside the predicted-class counts)
switch ( trueSignal )
{
case Buy :
m_trueBuyCount + + ;
break ;
case Sell :
m_trueSellCount + + ;
break ;
default :
m_trueNeutralCount + + ;
break ;
}
// OOS scoring used to happen right here, against whatever weights this bar's earlier
// feedForward (this pass) happened to be using - which for era 0 is the network's
// still-untrained cold-start state (100% Neutral - see the output-layer bias seed's
// declaration comment), and for every later era is last era's END-of-training state,
// never THIS era's. That silently gave every era's OOS score a full one-era lag behind
// its own training, and made era 0's OOS score meaningless by construction. OOS scoring
// now happens in its own pass (see m_isPass3Active's declaration comment), AFTER pass 2
// has actually trained on this era's IS data, against a fresh feedForward on each OOS
// bar rather than this scan's now-stale one.
if ( wouldQueue )
{
// Queue this bar for pass 2's shuffled backProp instead of training on it here,
// immediately, in strict chronological order - see m_isTrainQueue's declaration
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
// comment for the full rationale. The predicted-signal counts, the chart-marker draw,
// and the dForecast/dUndefine IS-accuracy update are all computed in pass 2 instead,
// against that bar's own freshly-recomputed confidence - see the matching block right
// after pass 2's Net.feedForward() call.
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
// EVERY BAR IS QUEUED EXACTLY ONCE. Class imbalance is corrected analytically inside
// the gradient by the logit-adjusted loss, not by duplicating minority bars here.
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
// The history is worth keeping, because it is why the data-level approach was
// abandoned rather than merely re-tuned. Four successive versions of oversampling all
// collapsed, in both directions:
// v1 uncapped replication x an independent loss weight (up to ~4.5x total) ->
// Buy-only collapse, OOS ~10%, IS error 0.37->0.57 in 4 eras.
// v2 capped the ratio before splitting it between the two -> mathematically the
// same total correction as pure loss weighting, which had already failed.
// v3 replication alone, capped at 3x against a ~5.3x imbalance -> Neutral collapse,
// Buy/Sell recall 0% for 6 straight eras (2026-07-18).
// v4 replication to ~90% parity (up to 28x) -> measured across six topologies on
// 2026-07-29, every model drove ONE direction to ~50% recall and abandoned the
// other, and which direction was arbitrary. One era in 1,301 cleared the floor.
// The through-line: replication makes Buy and Sell compete for the same replicated
// capacity, and Adam's mt/sqrt(vt) normalisation (Kingma & Ba 2015) is near-invariant
// to the gradient rescaling that the loss-weighted variants relied on. Per Buda, Maki
// & Mazurowski 2018, stacking data-level and cost-level corrections on one axis is not
// reliably additive - and the logit-adjusted loss replaces BOTH with a single
// correction that is provably consistent for balanced error.
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
// Pass 2 still Fisher-Yates shuffles the queue: chronological order correlates
// consecutive gradients, which is the same correlated-momentum overshoot documented at
// AI\Network.mqh's MAX_WEIGHT_DELTA comment. That reason is independent of replication
// and survives it.
//--- MINORITY REPLAY REMOVED 2026-07-31. Every bar is queued exactly once; class
//--- imbalance is corrected analytically in the gradient by the logit-adjusted loss
//--- (Menon et al. 2021) instead of by duplicating rare bars in the data. Stacking the
//--- two double-counts the same imbalance - Buda et al. 2018 - and the replay branch had
//--- in fact been gated OFF for the whole shipped configuration, so this is the code
//--- catching up with the behaviour rather than a change in it. Measured 2026-07-29
//--- across six topologies, replay made Buy and Sell compete for the same replicated
//--- capacity: every model drove ONE direction to ~50% recall and abandoned the other,
//--- and which direction was arbitrary. One era in 1,301 cleared the per-class floor.
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
int repCount = 1 ;
double perOccurrenceScale = 1.0 ;
//--- grow on demand (reserve keeps this amortized-rare) - the prealloc above is an
//--- estimate, and dropping overflow would silently starve exactly the minority
//--- classes the replication exists to protect
if ( m_isTrainQueueCount + repCount > ArraySize ( m_isTrainQueue ) )
{
int newQueueSize = m_isTrainQueueCount + repCount ;
ArrayResize ( m_isTrainQueue , newQueueSize , 16384 ) ;
ArrayResize ( m_isTrainQueueWeightScale , newQueueSize , 16384 ) ;
ArrayResize ( m_isTrainQueuePrimary , newQueueSize , 16384 ) ;
}
for ( int rep = 0 ; rep < repCount ; rep + + )
{
m_isTrainQueue [ m_isTrainQueueCount ] = i ;
m_isTrainQueueWeightScale [ m_isTrainQueueCount ] = perOccurrenceScale ;
//--- rep 0 is this bar's single "counts once" occurrence - see m_isTrainQueuePrimary.
//--- Every rep still trains; only the reported IS accuracy looks at this flag.
m_isTrainQueuePrimary [ m_isTrainQueueCount ] = ( rep = = 0 ) ;
m_isTrainQueueCount + + ;
}
}
}
stop = IsStopped ( ) | | m_trainingStopRequested ;
if ( ! stop & & i > 0 & & GetTickCount ( ) - chunkStartTick > = TRAIN_TIME_BUDGET_MS )
{
//--- yield: save exactly enough to resume this same era, mid-bar-loop, on the next call -
//--- see m_trainRunActive's declaration comment for why this must happen instead of
//--- letting one era (or the whole run) process synchronously to completion
m_resumeBars = bars ;
m_resumeTotalIter = totalIter ;
m_resumeOosCutoff = oosCutoff ;
m_resumeAddLoop = add_loop ;
m_resumeBarIndex = i - 1 ;
m_eraResumePending = true ;
// Save this model's own learning-rate trajectory back out of the shared global before
// yielding - see m_modelEta's declaration comment.
m_modelEta = eta ;
return ;
}
}
} // end if(!m_isPass2Active) - pass 1
//--- Pass 2: replay the bars pass 1 queued into m_isTrainQueue for backProp, in a freshly
//--- shuffled order - see m_isTrainQueue's declaration comment for the full rationale. Runs
//--- whenever pass 1 just finished (or we resumed straight into an already-active pass 2 - see
//--- m_isPass2Active's declaration comment); skipped on a stopped run, an era with no valid window
//--- at all (add_loop still false), or - critically - a resume into a still-unfinished pass 3 (see
//--- m_isPass2Done's declaration comment): without this last check, that resume would re-shuffle
//--- and replay the ENTIRE queue again from scratch every single call.
if ( ! stop & & add_loop & & ! m_isPass2Done )
{
if ( ! m_isPass2Active )
{
m_isPass2Active = true ;
m_isTrainCursor = 0 ;
// 2026-07-28: a "replay-only optimizer override" was removed from here. It captured every
// neuron's optimizer and forced the whole net to SGD for the duration of pass 2, on the
// rationale that oversampled minority bars should not "exploit the same Adam-style momentum
// path as the base training pass". But pass 2 IS the base training pass - it is the only place
// Net.backProp() is called during training at all (pass 1 only feeds forward and queues) - so
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
// the override applied to 100% of weight updates, not to some replay subset.
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// Adam's mt/vt were therefore never updated and its bias-correction step counter never
// advanced: TrainingOptimizer=ADAM was silently a no-op and the model trained purely on
// SGD+momentum at Adam's learning rate. It arrived with the DFA change set and was never part
// of any validated run. The optimizer the user selects is now the optimizer that runs.
// Fisher-Yates shuffle - a fresh random order every era, so Adam's momentum can't keep
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
// landing on the same contiguous same-class label run at the same point in the sequence every
// single era. Barrier labels make those runs LONGER than the old exact-pivot ones (adjacent
// bars share most of their forward window, so they usually resolve the same way), which makes
// the shuffle matter more here, not less. m_isTrainQueueWeightScale is swapped in lockstep - each
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// slot's stored per-occurrence weight (see the queueing block's oversampling comment) must
// stay attached to the same bar index it was computed for.
for ( int sIdx = m_isTrainQueueCount - 1 ; sIdx > 0 ; sIdx - - )
{
int sJ = MathRand ( ) % ( sIdx + 1 ) ;
int sTmp = m_isTrainQueue [ sIdx ] ;
m_isTrainQueue [ sIdx ] = m_isTrainQueue [ sJ ] ;
m_isTrainQueue [ sJ ] = sTmp ;
double sScaleTmp = m_isTrainQueueWeightScale [ sIdx ] ;
m_isTrainQueueWeightScale [ sIdx ] = m_isTrainQueueWeightScale [ sJ ] ;
m_isTrainQueueWeightScale [ sJ ] = sScaleTmp ;
//--- the primary flag must travel with its own slot too, or the "count this bar once"
//--- marker would end up attached to a different bar's occurrence - see m_isTrainQueuePrimary
bool sPrimTmp = m_isTrainQueuePrimary [ sIdx ] ;
m_isTrainQueuePrimary [ sIdx ] = m_isTrainQueuePrimary [ sJ ] ;
m_isTrainQueuePrimary [ sJ ] = sPrimTmp ;
}
}
for ( ; m_isTrainCursor < m_isTrainQueueCount ; m_isTrainCursor + + )
{
int qi = m_isTrainQueue [ m_isTrainCursor ] ;
fix: the sequence models were reading the window backwards
BuildFeatureWindow() replaces eight hand-rolled copies of the same loop
and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first,
because MQL5 timeseries indices run backwards and `r + b` with b ascending
walks into the past.
Harmless for PAI and CONV - a dense layer learns a weight per position
either way, a conv learns time-mirrored kernels. Not harmless for the
recurrent stacks:
- LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t.
- It writes output[] only when t == steps-1: the visible output IS the
last hidden state.
- c_t = f*c_{t-1} + i*g decays toward the start of the sequence.
lstm_seq_flowcheck.cpp measured block 0's influence on the output at
1.2e-2 of block T-1's, at the shipped forget bias of 1.0.
So the bar being PREDICTED sat at the far end of the decay and the output
was handed to the OLDEST bar in the window - the exact inverse of what the
window is for. ~80x backwards on LSTM and HYBRID, on all three tiers
(OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never
surfaced as a backend discrepancy.
This does not create edge - the MI diagnostics read at the noise floor
(p=0.4975) with a working positive control. It makes the one hypothesis
those diagnostics explicitly do NOT cover testable: they are marginal and
per-bar, and state they "cannot rule out one that only exists in
combination or across time". The sequence model is the instrument for
across-time structure and it has been crippled, so that hypothesis has
never been honestly tested.
Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and
its features, so a stale .nnw would load cleanly and run a model fitted to
one ordering against the other, silently. Re-keying every config is the
point, not collateral damage. FORCES A FULL RETRAIN.
Also: the now-relative bar caches are re-keyed on the two live paths.
EnsureBarCachesCapacity() was only ever called from training paths, but
once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar
to RefreshConvergedSignal() and Train() is never re-entered - so nothing
cleared the feature cache again for the life of the process. A chart that
trained to convergence kept replaying the rows computed for the last
training era's bar grid: the live signal froze at its convergence-time
value, and OnlineLearnStep() backpropped those stale features against
freshly resolved labels. Backtests were never affected (an inference-only
process never allocates the arrays, so every read recomputes).
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 18:28:44 -04:00
bool qWindowOk = BuildFeatureWindow ( qi ) ;
2026-08-02 01:09:18 -04:00
//--- A failed forward pass must NOT be followed by backProp() further down this block: the
//--- output layer would still hold the PREVIOUS sample's activations, so the update would be
//--- this bar's label against another bar's prediction - training on pure noise while every
//--- accuracy counter kept reporting normally.
bool qForwardOk = ( qWindowOk & & TempData . Total ( ) > = ( int ) m_historyBars * m_neuronsCount & &
Net . feedForward ( TempData ) ) ;
if ( qWindowOk & & ! qForwardOk & & ! forwardFailureReported )
{
forwardFailureReported = true ;
Print ( __FUNCTION__ + " : CNet::feedForward FAILED at era " + IntegerToString ( ( int ) m_eraCount ) +
" - this era's remaining samples are being skipped, not trained. A layer is refusing to "
" accept its own output (check the preceding BufferWrite/BufferRead lines for which "
" buffer, and see NormalizeHost in AI \\ NeuronBatchNorm.mqh for the batch-norm case). " ) ;
}
if ( qForwardOk )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
Net . getResults ( TempData ) ;
// Must go through ApplyClassificationSoftmax() (3-output case) before reading the
// per-class values below - Net.getResults() returns each output neuron's own independent
// SIGMOID activation (each already in [0,1] but NOT summing to 1 across the three), not a
// true class-conditional probability distribution; ApplyClassificationSoftmax() is what
// turns that into one (and is also what pass 1/3's displayNeuron0/1/2 already go through).
double qPrevSignal = ( m_outputNeuronsCount = = 3 ) ? ApplyClassificationSoftmax ( ) : TempData [ 0 ] ;
double pt0 = ( TempData . Total ( ) > 0 ) ? TempData [ 0 ] : 0.0 ;
double pt1 = ( TempData . Total ( ) > 1 ) ? TempData [ 1 ] : 0.0 ;
double pt2 = ( TempData . Total ( ) > 2 ) ? TempData [ 2 ] : 0.0 ;
bool qBuy = m_labelCacheHasValue [ qi ] ? m_labelCacheBuy [ qi ] : false ;
bool qSell = m_labelCacheHasValue [ qi ] ? m_labelCacheSell [ qi ] : false ;
ENUM_SIGNAL qTrueSignal = qBuy ? Buy : ( qSell ? Sell : Neutral ) ;
UpdateTrainingStatusLabel (
StringFormat ( " Training bar %d of %d -> %.2f%% (shuffled) " , m_isTrainCursor + 1 , m_isTrainQueueCount , ( double ) ( m_isTrainCursor + 1.0 ) / MathMax ( m_isTrainQueueCount , 1 ) * 100 ) ,
pt0 , pt1 , pt2 , qPrevSignal ) ;
//--- Predicted-signal tally, chart marker, and IS-accuracy stat that pass 1 used to compute
//--- from its own (now-removed) redundant feedForward on this same bar - see pass 1's
//--- wouldQueue comment. Uses THIS feedForward's result (the only one this bar gets), so
//--- these now reflect the model's state as of this bar's own turn in the shuffled replay
//--- (post any earlier-shuffled bar's backProp this era), not a separate pre-training
//--- snapshot - matching how a standard shuffled-epoch SGD run reports running training
//--- accuracy during the epoch rather than in a discarded pre-epoch dry run.
switch ( DoubleToSignal ( qPrevSignal ) )
{
case Buy :
m_countBuySignals + + ;
break ;
case Sell :
m_countSellSignals + + ;
break ;
default :
m_countNeutralSignals + + ;
break ;
}
datetime qBarTime = m_Time . GetData ( qi ) ;
// NMS on: record only (the era-end sweep renders); off: draw inline. See pass 1's note.
if ( m_signalClusterWindow > 0 )
{
if ( qi < ArraySize ( m_arrowSignalCache ) )
m_arrowSignalCache [ qi ] = qPrevSignal ;
}
else
if ( DoubleToSignal ( qPrevSignal ) = = Neutral )
DeleteObject ( qBarTime ) ;
else
DrawObject ( qBarTime , qPrevSignal , m_High . GetData ( qi ) , m_Low . GetData ( qi ) ) ;
bool qClassified = ( DoubleToSignal ( qPrevSignal ) = = Buy | | DoubleToSignal ( qPrevSignal ) = = Sell | | DoubleToSignal ( qPrevSignal ) = = Neutral ) ;
if ( qClassified )
{
bool isHit = ( DoubleToSignal ( qPrevSignal ) = = qTrueSignal ) ;
if ( isHit )
dForecast + = ( 100 - dForecast ) / Net . recentAverageSmoothingFactor ;
else
dForecast - = dForecast / Net . recentAverageSmoothingFactor ;
dUndefine - = dUndefine / Net . recentAverageSmoothingFactor ;
//--- Compounded, persistent DIRECTIONAL win-rate: count only bars the model actually called
//--- Buy or Sell (a Neutral "no trade" call is neither a win nor a loss), so this tracks the
//--- accuracy of its directional signals rather than the Neutral-inflated all-class rate.
//--- ...and count each BAR once, not each oversampled OCCURRENCE (m_isTrainQueuePrimary):
//--- the queue duplicates minority bars up to ~21x, so counting every occurrence scored this
//--- metric over a ~58%-directional set while its OOS counterpart scored the real ~6%
//--- distribution - two numbers that look comparable, aren't, and made a healthy run read as
//--- severe overfitting. See m_isTrainQueuePrimary for the worked example.
ENUM_SIGNAL qPred = DoubleToSignal ( qPrevSignal ) ;
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
if ( m_isTrainQueuePrimary [ m_isTrainCursor ] & & ( qPred = = Buy | | qPred = = Sell ) )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
m_cumIsTotal + + ;
if ( isHit )
m_cumIsCorrect + + ;
}
}
else
if ( qBuy & & qSell )
dUndefine + = ( 100 - dUndefine ) / Net . recentAverageSmoothingFactor ;
TempData . Clear ( ) ;
if ( m_outputNeuronsCount = = 1 )
TempData . Add ( qBuy & & ! qSell ? 1 : ! qBuy & & qSell ? -1 : 0 ) ;
else
if ( m_outputNeuronsCount = = 3 )
{
TempData . Add ( qBuy ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW ) ;
TempData . Add ( qSell ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW ) ;
TempData . Add ( ( ! qBuy & & ! qSell ) ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW ) ;
}
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
// Per-slot weight from m_isTrainQueueWeightScale[m_isTrainCursor], decided once at queue
// time in pass 1. Currently always 1.0: class imbalance is corrected analytically inside
// the gradient by the logit-adjusted loss, so there is no per-sample reweighting left to
// apply here at all. Kept as a real per-slot value rather than a literal 1.0 inline so a
// future supplemental weight can be reintroduced without re-touching the queueing or
// shuffle code.
//--- FOCAL-LOSS MODULATION REMOVED 2026-07-31. It multiplied this weight by (1-pt)^gamma,
//--- a second correction on the same axis as the logit adjustment - the stacking failure
//--- Buda et al. 2018 describes and this file already cited in two other places. It was
//--- running at an eighth strength (gamma * 0.125), damped by the replay toggle, for a
//--- replay path that the adjusted loss had already switched off - so the damping was
//--- calibrated against a mechanism that was not running. See the class-imbalance audit in
//--- Variables\Inputs.mqh.
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
double qSampleWeight = m_isTrainQueueWeightScale [ m_isTrainCursor ] ;
Net . backProp ( TempData , qSampleWeight ) ;
}
if ( m_isTrainCursor + 1 < m_isTrainQueueCount & & GetTickCount ( ) - chunkStartTick > = TRAIN_TIME_BUDGET_MS )
{
//--- yield: save enough to resume PASS 2 mid-queue on the next call - m_isPass2Active
//--- and m_isTrainCursor (both members) carry the actual resume position; bars/oosCutoff/
//--- add_loop are stashed the same way pass 1 already does, since era-end logic just
//--- below still needs them once pass 2 finishes.
m_resumeBars = bars ;
m_resumeTotalIter = totalIter ;
m_resumeOosCutoff = oosCutoff ;
m_resumeAddLoop = add_loop ;
m_resumeBarIndex = i ;
m_eraResumePending = true ;
m_modelEta = eta ;
return ;
}
}
m_isPass2Active = false ;
m_isPass2Done = true ;
}
//--- Pass 3: OOS scoring, chronological, AFTER pass 2 has actually trained on this era's IS data -
//--- see m_isPass3Active's declaration comment for why this can no longer happen inline during
//--- pass 1's scan.
if ( ! stop & & add_loop )
{
if ( ! m_isPass3Active )
{
m_isPass3Active = true ;
m_oosScoreStartIndex = ( int ) MathMin ( oosCutoff - 1 , bars - MathMax ( m_historyBars , 0 ) - 2 ) ;
m_oosScoreIndex = m_oosScoreStartIndex ;
for ( int rn = 0 ; rn < 3 ; rn + + )
{
m_oosOutMin [ rn ] = DBL_MAX ;
m_oosOutMax [ rn ] = - DBL_MAX ;
}
m_oosOutSpreadSum = 0.0 ;
m_oosOutCount = 0 ;
}
for ( ; m_oosScoreIndex > = 2 ; m_oosScoreIndex - - )
{
int oi = m_oosScoreIndex ;
if ( ! ( oi < ( int ) ( bars - MathMax ( m_historyBars , 0 ) - 1 ) & & m_Time . GetData ( oi ) > dtStudied ) )
continue ;
fix: the sequence models were reading the window backwards
BuildFeatureWindow() replaces eight hand-rolled copies of the same loop
and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first,
because MQL5 timeseries indices run backwards and `r + b` with b ascending
walks into the past.
Harmless for PAI and CONV - a dense layer learns a weight per position
either way, a conv learns time-mirrored kernels. Not harmless for the
recurrent stacks:
- LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t.
- It writes output[] only when t == steps-1: the visible output IS the
last hidden state.
- c_t = f*c_{t-1} + i*g decays toward the start of the sequence.
lstm_seq_flowcheck.cpp measured block 0's influence on the output at
1.2e-2 of block T-1's, at the shipped forget bias of 1.0.
So the bar being PREDICTED sat at the far end of the decay and the output
was handed to the OLDEST bar in the window - the exact inverse of what the
window is for. ~80x backwards on LSTM and HYBRID, on all three tiers
(OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never
surfaced as a backend discrepancy.
This does not create edge - the MI diagnostics read at the noise floor
(p=0.4975) with a working positive control. It makes the one hypothesis
those diagnostics explicitly do NOT cover testable: they are marginal and
per-bar, and state they "cannot rule out one that only exists in
combination or across time". The sequence model is the instrument for
across-time structure and it has been crippled, so that hypothesis has
never been honestly tested.
Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and
its features, so a stale .nnw would load cleanly and run a model fitted to
one ordering against the other, silently. Re-keying every config is the
point, not collateral damage. FORCES A FULL RETRAIN.
Also: the now-relative bar caches are re-keyed on the two live paths.
EnsureBarCachesCapacity() was only ever called from training paths, but
once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar
to RefreshConvergedSignal() and Train() is never re-entered - so nothing
cleared the feature cache again for the life of the process. A chart that
trained to convergence kept replaying the rows computed for the last
training era's bar grid: the live signal froze at its convergence-time
value, and OnlineLearnStep() backpropped those stale features against
freshly resolved labels. Backtests were never affected (an inference-only
process never allocates the arrays, so every read recomputes).
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 18:28:44 -04:00
bool oWindowOk = BuildFeatureWindow ( oi ) ;
2026-08-02 01:09:18 -04:00
//--- Same guard as pass 2, and it matters more here: OOS accuracy is what checkpoint selection
//--- and the plateau ladder's auto-deploy both rank on, so scoring a stale forward pass would
//--- not just be wrong, it would be wrong in the one number that decides which model ships.
//--- A skipped bar simply isn't counted; it never becomes a hit or a miss.
bool oForwardOk = ( oWindowOk & & TempData . Total ( ) > = ( int ) m_historyBars * m_neuronsCount & &
Net . feedForward ( TempData ) ) ;
if ( oWindowOk & & ! oForwardOk & & ! forwardFailureReported )
{
forwardFailureReported = true ;
Print ( __FUNCTION__ + " : CNet::feedForward FAILED during OOS scoring at era " +
IntegerToString ( ( int ) m_eraCount ) + " - affected bars are excluded from the OOS "
" accuracy rather than scored against a stale prediction. " ) ;
}
if ( oForwardOk )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
Net . getResults ( TempData ) ;
// Raw output stats MUST be captured here, before ApplyClassificationSoftmax() overwrites
// TempData[0..2] in place with the softmax probabilities - see m_oosOutMin's declaration
// comment for what these feed.
if ( m_outputNeuronsCount = = 3 & & TempData . Total ( ) > = 3 )
{
double rawHi = - DBL_MAX , rawLo = DBL_MAX ;
for ( int rn = 0 ; rn < 3 ; rn + + )
{
double rv = TempData . At ( rn ) ;
if ( rv < m_oosOutMin [ rn ] )
m_oosOutMin [ rn ] = rv ;
if ( rv > m_oosOutMax [ rn ] )
m_oosOutMax [ rn ] = rv ;
rawHi = MathMax ( rawHi , rv ) ;
rawLo = MathMin ( rawLo , rv ) ;
}
m_oosOutSpreadSum + = rawHi - rawLo ;
m_oosOutCount + + ;
}
double oPrevSignal = ( m_outputNeuronsCount = = 3 ) ? ApplyClassificationSoftmax ( ) : TempData [ 0 ] ;
double oDeploySignal = oPrevSignal ;
if ( m_outputNeuronsCount = = 3 )
oDeploySignal = AdjustedSignalFromSoftmax ( ) ;
double oNeuron0 = ( TempData . Total ( ) > 0 ) ? TempData [ 0 ] : 0.0 ;
double oNeuron1 = ( TempData . Total ( ) > 1 ) ? TempData [ 1 ] : 0.0 ;
double oNeuron2 = ( TempData . Total ( ) > 2 ) ? TempData [ 2 ] : 0.0 ;
bool oBuy = m_labelCacheHasValue [ oi ] ? m_labelCacheBuy [ oi ] : false ;
bool oSell = m_labelCacheHasValue [ oi ] ? m_labelCacheSell [ oi ] : false ;
ENUM_SIGNAL oTrueSignal = oBuy ? Buy : ( oSell ? Sell : Neutral ) ;
UpdateTrainingStatusLabel (
StringFormat ( " Scoring OOS bar %d of %d -> %.2f%% (post-training) " , m_oosScoreStartIndex - m_oosScoreIndex + 1 , m_oosScoreStartIndex + 1 ,
( double ) ( m_oosScoreStartIndex - m_oosScoreIndex + 1.0 ) / MathMax ( m_oosScoreStartIndex + 1 , 1 ) * 100 ) ,
oNeuron0 , oNeuron1 , oNeuron2 , oDeploySignal ) ;
// Held-out bar: score the model's freshly-trained-this-era forecast against the actual
// outcome without learning from it - keeps the OOS accuracy an honest overfitting signal.
bool oClassified = ( DoubleToSignal ( oPrevSignal ) = = Buy | | DoubleToSignal ( oPrevSignal ) = = Sell | | DoubleToSignal ( oPrevSignal ) = = Neutral ) ;
if ( oClassified )
{
m_oosSamples + + ;
m_oosConfidenceSum + = MathAbs ( oPrevSignal ) ;
if ( dOosError < 0 )
dOosError = 0 ;
bool hit = ( DoubleToSignal ( oPrevSignal ) = = oTrueSignal ) ;
//--- Compounded, persistent DIRECTIONAL win-rate: count only bars the model actually called
//--- Buy or Sell (Neutral "no trade" calls aren't wins or losses), so this reflects the
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- accuracy of its directional signals, not the Neutral-inflated all-class rate.
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
ENUM_SIGNAL oPred = DoubleToSignal ( oPrevSignal ) ;
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
if ( oPred = = Buy | | oPred = = Sell )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
m_cumOosTotal + + ;
if ( hit )
m_cumOosCorrect + + ;
}
// Per-class confusion counts, used for the Buy/Sell recall convergence gate below
switch ( oTrueSignal )
{
case Buy :
m_oosBuyTotal + + ;
if ( hit )
m_oosBuyHits + + ;
break ;
case Sell :
m_oosSellTotal + + ;
if ( hit )
m_oosSellHits + + ;
break ;
default :
m_oosNeutralTotal + + ;
if ( hit )
m_oosNeutralHits + + ;
break ;
}
// Same confusion counts keyed by what the model actually PREDICTED this bar, not the
// true label - see m_oosBuyPredicted's declaration comment for why recall alone can
// hide an over-firing class.
switch ( DoubleToSignal ( oPrevSignal ) )
{
case Buy :
m_oosBuyPredicted + + ;
if ( hit )
m_oosBuyPredictedHits + + ;
break ;
case Sell :
m_oosSellPredicted + + ;
if ( hit )
m_oosSellPredictedHits + + ;
break ;
default :
m_oosNeutralPredicted + + ;
if ( hit )
m_oosNeutralPredictedHits + + ;
break ;
}
// Live-decision precision: scores the bars on which the deployed EA would actually cast a
// directional vote, using the prior-corrected (logit-adjusted) posterior - see
// AdjustedSignalFromSoftmax()/RefreshLatestSignal(). The recall/argmax-precision above stay
// on the raw argmax (the model's intrinsic class separation, which the convergence gate
// needs); THIS scores what trades live, so the panel's live precision number is the
// precision a buyer gets forward. TempData still holds this bar's raw softmax probs
// (nothing overwrote them since ApplyClassificationSoftmax above), so the adjustment reads
// them directly. Neutral picks aren't counted - they cast no vote.
// No confidence-floor term any more: with the floor removed, EVERY non-Neutral adjusted
// decision casts a vote (at its tier weight), so any threshold here would score a
// different population than the one that actually votes. Whether a given vote goes on to
// OPEN a position additionally depends on Min_Vote_Open versus the AVERAGE across all
// voting filters, which this per-bar training-time scorer has no visibility of - so this
// stays the honest "would have voted, and was it right" measure rather than pretending to
// model the aggregate.
if ( m_outputNeuronsCount = = 3 )
{
double adjSig = AdjustedSignalFromSoftmax ( ) ;
ENUM_SIGNAL adjEnum = DoubleToSignal ( adjSig ) ;
if ( adjEnum ! = Neutral )
{
bool fireHit = ( adjEnum = = oTrueSignal ) ;
feat(ai): measure precision per confidence tier; fix stale metric labels
Two things the 2026-07-30 run exposed.
1. Every user-facing message still called the selection metric "balanced
accuracy". It has ranked on directional precision since a142749, so
"CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5%
PRECISION as if it were macro-recall, while the same era logged an
actual balanced accuracy of 49%. Two different numbers under one
name, in the line that announces a deploy. Relabelled at every site,
including the stage-3 refusal, which still described the per-class
recall floor that stopped being the gate.
2. Precision is now bucketed by confidence tier and logged per era,
both per-tier and cumulatively from each tier upward:
| tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ...
The per-tier number says whether confidence is calibrated to
correctness at all; if it does not rise T0->T3, raising the floor
buys nothing and that is the finding. The ">=" number is what a floor
would actually deliver, with its fire count, so the coverage cost is
visible in the same line. Tier weights are 25/50/75/100, so for an
AI-only config Min_Vote_Open maps straight across: 50 = ">=T1",
75 = ">=T2", 100 = ">=T3".
Bucketing happens at the existing live-fired accounting site, so it
measures exactly the population that trades - not the raw argmax.
Both builds compile 0 errors, 0 warnings. No retrain needed for either.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
//--- Bucket the same fire by confidence tier - see m_oosTierFired. Safe to call here
//--- and nowhere earlier: ConfidenceTier() reads the net's CURRENT outputs, which is
//--- exactly the bar AdjustedSignalFromSoftmax() just scored.
int fireTier = ConfidenceTier ( ) ;
if ( fireTier > = 0 & & fireTier < 4 )
{
m_oosTierFired [ fireTier ] + + ;
if ( fireHit )
m_oosTierHits [ fireTier ] + + ;
}
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
if ( adjEnum = = Buy )
{
m_oosBuyFired + + ;
if ( fireHit )
m_oosBuyFiredHits + + ;
}
else
{
m_oosSellFired + + ;
if ( fireHit )
m_oosSellFiredHits + + ;
}
}
}
if ( hit )
{
dOosForecast + = ( 100 - dOosForecast ) / Net . recentAverageSmoothingFactor ;
dOosError - = dOosError / Net . recentAverageSmoothingFactor ;
}
else
{
dOosForecast - = dOosForecast / Net . recentAverageSmoothingFactor ;
dOosError + = ( 100 - dOosError ) / Net . recentAverageSmoothingFactor ;
}
}
// Mirror pass 1's chart annotation for this (OOS) bar, now using post-training weights
// instead of pass 1's pre-training snapshot - last write wins on the shared per-bar-time
// object, so this supersedes pass 1's earlier draw for the same bar with the correct one.
m_lastBarTime = m_Time . GetData ( oi ) ;
if ( oi > 0 )
{
// NMS on: record only (the era-end sweep renders); off: draw inline. See pass 1's note.
if ( m_signalClusterWindow > 0 )
{
if ( oi < ArraySize ( m_arrowSignalCache ) )
m_arrowSignalCache [ oi ] = oDeploySignal ;
}
else
if ( DoubleToSignal ( oDeploySignal ) = = Neutral )
DeleteObject ( m_lastBarTime ) ;
else
DrawObject ( m_lastBarTime , oDeploySignal , m_High . GetData ( oi ) , m_Low . GetData ( oi ) ) ;
}
}
if ( m_oosScoreIndex - 1 > = 2 & & GetTickCount ( ) - chunkStartTick > = TRAIN_TIME_BUDGET_MS )
{
//--- yield: save enough to resume PASS 3 mid-walk on the next call - m_isPass3Active and
//--- m_oosScoreIndex (both members) carry the actual resume position.
m_resumeBars = bars ;
m_resumeTotalIter = totalIter ;
m_resumeOosCutoff = oosCutoff ;
m_resumeAddLoop = add_loop ;
m_resumeBarIndex = i ;
m_eraResumePending = true ;
m_modelEta = eta ;
return ;
}
}
m_isPass3Active = false ;
//--- Pass 3 done => every scored bar's prediction is now in m_arrowSignalCache. Collapse each
//--- same-direction cluster to its earliest bar so the chart shows one arrow per real turn.
PruneDirectionalClusters ( bars ) ;
}
//--- Diagnostic recall snapshot for the periodic progress log further below - populated inside
//--- the m_oosSamples>0 recall-gate block when this era actually computes it; stays -1 ("n/a"
//--- in the log) on eras that don't (era 0, or a stopped/cap-hit era).
int logBuyRecallPct = -1 , logSellRecallPct = -1 , logNeutralRecallPct = -1 ;
//--- Balanced accuracy (macro-recall) this era, surfaced in the log so the metric the checkpoint
//--- is now selected on is visible - see m_bestBalancedOos. -1 ("n/a") on eras that don't score.
int logBalancedAccPct = -1 ;
feat(ai): rank checkpoints on directional precision, not balanced accuracy
Balanced accuracy is maximized by exactly the model this system must never
deploy. Measured frontier at fixed signal strength, base rate 6.1%:
tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0%
tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3%
tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5%
It rises monotonically as the model calls MORE and is right LESS, because
two of its three terms are directional recalls that a call-everything model
drives to ~95%, while the Neutral term it sacrifices counts for only a
third. The 2026-07-29 run landed exactly there: balanced 58-64% while
calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base
rate. Only the per-class recall floor stopped those deploying - a guard
doing the job the objective should have been doing - and that same guard
also rejected the genuinely useful sparse-but-precise checkpoints.
Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how
many were right. That is what a trading edge is. Two anti-degenerate floors
bracket it, since precision alone is trivially maximized by calling almost
nothing: coverage must reach a fraction of the true directional base rate
(derived, not configured - it adapts to any symbol/timeframe/label rule),
and precision must at least beat that base rate.
Against the same frontier the deploy order inverts from
tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first)
to
tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage)
Balanced accuracy is kept in the log as a diagnostic and marked as such, so
a run where the two disagree - the signature of an over-caller - is visible
at a glance. MinRecall no longer decides what ships; it now only drives the
diagnostic recall line and is a candidate for removal.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
int logCoveragePct = -1 ;
int logDirPrecPct = -1 ;
fix: refuse invalid SL/TP, fix the unreachable deploy floor, scale the horizon
Three defects found by reading the 2026-08-01 training logs, all of which
only became visible because the relabel made the numbers mean something.
1. A STALE ENUM TRAINED FOUR MODELS ON THE WRONG TARGET.
`OnInit: trade settings snapshot - SL_Mode=1 TP_Mode=-101`
-101 was TP_PREV_SWING, deleted from TAKE_PROFIT_MODE on 2026-07-31 in
7eb48f5. MetaTrader does not validate a saved enum input against the
enum's current members, so charts saved before that kept the old
integer. BarrierMultiples()'s `if(tpMult <= 0.0) tpMult = slMult;`
then quietly turned it into a 1:1 barrier, and all four topologies
trained ~250 eras against a strategy nobody selected - while the log
reported "target 1.00*ATR" as though it were configured.
Since the relabel these two inputs ARE the label definition, so this
is not a bad trade setting, it is a wrong dataset. ValidateBarrier-
Inputs() now refuses to start (INIT_FAILED + Alert + an explicit fix)
on any value that is not an enum member. Members are enumerated rather
than range-checked because both enums are sparse and carry negative
sentinels, so no min/max test can tell a legal value from a deleted
one - which is the entire failure mode. The fallback survives as
belt-and-braces but now announces itself: a fallback that cannot say
it fired is indistinguishable from correct behaviour.
2. THE DEPLOYABILITY FLOOR BECAME MATHEMATICALLY UNREACHABLE.
`tradeableOK` required `dirPrecPct >= baseRatePct`, where baseRatePct
is Buy+Sell as a share of all bars. At the old exact-pivot target that
was ~6%, so "beat the base rate" read as "beat chance" and the test
looked sound. Triple-barrier labels put it at ~83%, so the gate now
demanded 83% directional precision - impossible by construction.
Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing
safe to deploy" at a perfectly healthy 43-45% precision, with no
checkpoint able to ship however good it got.
Replaced with ZERO-SKILL precision, max(Buy,Sell)/allBars: exactly the
score of the degenerate always-call-one-direction model this floor
exists to reject. Correct at any base rate - ~43% on the current
labels, ~3% on the old rare-pivot ones. The era line now prints
"(chance N%, edge +Mpp)" beside the selection score, because 44%
precision is excellent against a 3% chance level and worthless against
a 43% one, and reading the first as the second is what made tonight's
run look better than it was.
3. THE HORIZON IGNORED THE BARRIER GEOMETRY.
ComputeBarrierHorizonBars() returned the median ZigZag leg, which
measures how long a ~1 ATR move takes and says nothing about how long
the CONFIGURED barrier needs. First-passage time out of [-m,+k] scales
with m*k, so a 1:3 barrier takes ~3x as long as 1:1; the unscaled
horizon would have timed out most 1:3 trades and pushed Neutral
straight back up, re-creating the imbalance the relabel removes.
Now multiplied by slMult*tpMult, calibrated against a real measurement
rather than assumed: the accidental 1:1 run resolved at horizon 12 with
only 16.7% timeouts, so the swing median is the right scale at m*k=1.
Verifiable, not just asserted: the prebuild now counts barriers that
ended on the VERTICAL barrier and reports them as a share of Neutral.
Neutral conflates "timed out" with "stopped out" and only the first
indicts the horizon.
Both builds compile 0 errors / 0 warnings. Forces a retrain - correcting
TP_Mode re-keys the fingerprint (|TB:1:-101 -> |TB:1:3), which is right:
no existing model was trained on the intended target.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:30:49 -04:00
//--- Zero-skill precision for this era's label mix - see chancePrecPct. Logged beside the selection
//--- score because the raw precision number is meaningless without it: 44% is excellent against a
//--- 3% chance level and worthless against a 43% one, and the whole 2026-08-01 confusion was
//--- reading the first as if it were the second.
int logChancePrecPct = -1 ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- Predicted-rate (of all OOS bars this era, how often the model called this class at all) and
//--- precision (of the calls it made, how many were right) for Buy/Sell - m_oosBuyPredicted/
//--- m_oosSellPredicted (see that member's declaration comment) were already being tracked for
//--- exactly this but never surfaced anywhere. A recall-only view can't tell "the model never once
//--- calls Sell" (predicted rate stuck at 0%) apart from "the model calls Sell plenty but always on
//--- the wrong bars" (predicted rate healthy, precision near 0%) - both show up identically as 0%
//--- Sell recall, but point at completely different problems (a suppressed/dead output vs. a
//--- miscalibrated decision boundary), so this splits them out.
int logBuyPredPct = -1 , logSellPredPct = -1 , logBuyPrecPct = -1 , logSellPrecPct = -1 ;
//--- Live-fired precision (%) per direction this era - the precision on just the bars that cleared
//--- the confidence floor under the live/prior-corrected decision, i.e. what would actually trade.
int logBuyFiredPrecPct = -1 , logSellFiredPrecPct = -1 ;
bool shouldLogProgress = false ;
//--- era complete (ran out of bars) or a stop was requested mid-era
if ( add_loop )
{
m_eraCount + + ;
m_erasSinceCooldown + + ;
//--- EMA shadow-weight deployment: blend the shadow a small step (SHADOW_WEIGHT_TAU) toward
//--- Net's just-updated weights, every era - see m_shadowNet's declaration comment. Must run
//--- here, inside the era loop, not just once at Train()-end: the whole point is damping the
//--- WITHIN-run oscillation (era-to-era whipsaw), which a single end-of-run blend would miss
//--- entirely.
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
EnsureShadowNet ( ) ;
if ( CheckPointer ( m_shadowNet ) ! = POINTER_INVALID )
m_shadowNet . BlendWeightsFrom ( Net , SHADOW_WEIGHT_TAU ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- Status-label progress is invisible with no chart (headless/optimization runs), and even in
//--- visual mode a long training run can otherwise look "stuck" for a long time with no
//--- Journal output at all - log progress at most every ~5s (real wall-clock, not simulated
//--- time) so an operator can tell it's actively working, not hung. The actual Print() is
//--- deferred past the recall-gate block below (see logBuyRecallPct etc.) so this line can
//--- show per-class OOS recall - once OOS accuracy alone clears the target, recall is the
//--- most common thing still silently blocking convergence, and previously had no visibility
//--- outside of a regression event.
static uint lastProgressLogTick = 0 ;
uint nowTick = GetTickCount ( ) ;
shouldLogProgress = ( nowTick - lastProgressLogTick > = 5000 ) ;
if ( shouldLogProgress )
lastProgressLogTick = nowTick ;
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- Era cap. There used to be a second, much smaller cap here for throwaway auto-tune
//--- candidates; the filter tuner does not train candidates at all, so only the real one remains.
int effectiveEraCap = m_maxErasPerRun ;
2026-07-31 14:33:29 -04:00
//--- PLATEAU LADDER, terminal stage: training stopped improving and both escape attempts (two
//--- learning-rate warm restarts) failed to find anything better - see the ladder in the era-end block
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- below, which is what raised m_plateauStage this far and already logged why. This is the
//--- normal, expected way a run finishes now that there is no absolute accuracy target to hit:
//--- it trains until it genuinely stops getting better, then deploys its best checkpoint.
//--- Same mechanism as the operator's "No" answer at the era cap (see that branch's comments for
//--- why m_trainingComplete is set here and why m_trainingStopRequested deliberately is NOT):
//--- stop ends this era loop, FinalizeTrainRun() then restores and deploys the best checkpoint.
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
if ( m_plateauStage > = PLATEAU_STAGE_DEPLOY & & m_bestPassedRecall & & m_haveOosCheckpoint )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
stop = true ;
m_trainingComplete = true ;
}
else if ( effectiveEraCap > 0 & & m_erasSinceCooldown > = effectiveEraCap )
{
//--- Era cap reached without converging: ask the operator whether to keep training or
//--- deploy the best checkpoint and stop (see PromptContinuePastEraCap / m_maxErasPerRun).
if ( PromptContinuePastEraCap ( dOosForecast ) )
{
m_erasSinceCooldown = 0 ; // keep training - reset the cap window
feat(ai): measure precision per confidence tier; fix stale metric labels
Two things the 2026-07-30 run exposed.
1. Every user-facing message still called the selection metric "balanced
accuracy". It has ranked on directional precision since a142749, so
"CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5%
PRECISION as if it were macro-recall, while the same era logged an
actual balanced accuracy of 49%. Two different numbers under one
name, in the line that announces a deploy. Relabelled at every site,
including the stage-3 refusal, which still described the per-class
recall floor that stopped being the gate.
2. Precision is now bucketed by confidence tier and logged per era,
both per-tier and cumulatively from each tier upward:
| tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ...
The per-tier number says whether confidence is calibrated to
correctness at all; if it does not rise T0->T3, raising the floor
buys nothing and that is the finding. The ">=" number is what a floor
would actually deliver, with its fire count, so the coverage cost is
visible in the same line. Tier weights are 25/50/75/100, so for an
AI-only config Min_Vote_Open maps straight across: 50 = ">=T1",
75 = ">=T2", 100 = ">=T3".
Bucketing happens at the existing live-fired accounting site, so it
measures exactly the population that trades - not the raw argmax.
Both builds compile 0 errors, 0 warnings. No retrain needed for either.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
Print ( ID + " : hit the " + IntegerToString ( m_maxErasPerRun ) + " -era cap (best dir-precision " + DoubleToString ( m_bestBalancedOos , 1 ) + " %, blended OOS " + DoubleToString ( dOosForecast , 1 ) + " %) - CONTINUING training by operator choice. " ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
else
{
//--- stop: end THIS era loop now; FinalizeTrainRun (reached via the stop path below,
//--- because stop==true) deploys the best checkpoint. Deliberately do NOT set
//--- m_trainingStopRequested here: m_trainingComplete alone already routes every later
//--- tick to RefreshConvergedSignal (see ScheduleTrainingIfNeeded's if-branch precedence),
//--- so training never re-arms - and leaving m_trainingStopRequested false lets the deployed
//--- model run live inference AND online continual learning IN-SESSION, exactly like a
//--- normal-convergence deploy (which never sets it either). A panel Stop (StopTraining())
//--- still sets it and halts everything, including online learning - that distinction is
//--- preserved. See OnlineLearnStep()'s gate.
stop = true ;
//--- Operator DELIBERATELY chose to deploy this best checkpoint as the final model. That's a
//--- terminal decision and must be PERSISTED as such: mark it complete so a later reload
//--- (chart restart OR strategy tester) runs inference instead of silently resuming a full
//--- training run. This is the terminal-deploy path; a mid-training Stop click
//--- (StopTraining()) leaves m_trainingComplete false on purpose so that genuinely-
//--- interrupted run does resume. Note the m_trainingComplete=(m_objectiveMet&&m_oosStable)
//--- line below is inside if(!stop), so it can't clobber this back to false on this path.
m_trainingComplete = true ;
feat(ai): measure precision per confidence tier; fix stale metric labels
Two things the 2026-07-30 run exposed.
1. Every user-facing message still called the selection metric "balanced
accuracy". It has ranked on directional precision since a142749, so
"CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5%
PRECISION as if it were macro-recall, while the same era logged an
actual balanced accuracy of 49%. Two different numbers under one
name, in the line that announces a deploy. Relabelled at every site,
including the stage-3 refusal, which still described the per-class
recall floor that stopped being the gate.
2. Precision is now bucketed by confidence tier and logged per era,
both per-tier and cumulatively from each tier upward:
| tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ...
The per-tier number says whether confidence is calibrated to
correctness at all; if it does not rise T0->T3, raising the floor
buys nothing and that is the finding. The ">=" number is what a floor
would actually deliver, with its fire count, so the coverage cost is
visible in the same line. Tier weights are 25/50/75/100, so for an
AI-only config Min_Vote_Open maps straight across: 50 = ">=T1",
75 = ">=T2", 100 = ">=T3".
Bucketing happens at the existing live-fired accounting site, so it
measures exactly the population that trades - not the raw argmax.
Both builds compile 0 errors, 0 warnings. No retrain needed for either.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
Print ( ID + " : hit the " + IntegerToString ( m_maxErasPerRun ) + " -era cap before the plateau ladder finished (best dir-precision " + DoubleToString ( m_bestBalancedOos , 1 ) + " %, blended OOS " + DoubleToString ( dOosForecast , 1 ) + " %) - operator chose to DEPLOY the best checkpoint as final (marked complete; reloads will run inference, not retrain). Reaching this cap now means the run was still finding new bests, or never cleared the per-class recall floor (need >= " + IntegerToString ( m_minDirectionalRecallPct ) + " % each) - raise the era cap for the former, relax MinRecall/SwingConfirmationBars for the latter. " ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
}
}
if ( ! stop )
{
dError = Net . getRecentAverageError ( ) ;
if ( add_loop )
{
if ( m_oosSamples > 0 )
{
// Confidence calibration (classification head only - see m_confidenceCalScale's
// declaration comment): compare this era's actual OOS accuracy against the average
// confidence magnitude the model claimed, EMA-blend the resulting scale into
// m_confidenceCalScale so SignedAIConfidence() reports something closer to a real
// probability instead of the raw, uncalibrated softmax value.
if ( m_outputNeuronsCount = = 3 & & m_oosConfidenceSum > 0.0 )
{
double empiricalAccuracy = ( double ) ( m_oosBuyHits + m_oosSellHits + m_oosNeutralHits ) / m_oosSamples ;
double avgClaimedConfidence = m_oosConfidenceSum / m_oosSamples ;
double eraScale = MathMax ( 0.3 , MathMin ( 1.5 , empiricalAccuracy / avgClaimedConfidence ) ) ;
m_confidenceCalScale + = ( eraScale - m_confidenceCalScale ) / Net . recentAverageSmoothingFactor ;
}
// Per-class recall gate, symmetric across all three classes: a model that "wins" on
// blended dOosForecast purely by calling everything Neutral (or, just as biased, by
// over-calling Buy/Sell at Neutral's expense) would still pass a plain accuracy check -
// require Buy, Sell, AND Neutral OOS recall to each individually clear
// m_minDirectionalRecallPct so the network can't converge while biased toward any one
// output. A class with FEWER than MIN_OOS_CLASS_SAMPLES_FOR_GATE true OOS samples this
// era doesn't block (recallPct == -1 => treated as passing) so a thin OOS window doesn't
// deadlock convergence early in a run. Computed BEFORE the checkpoint/eta-decay block
// below (not just the final m_objectiveMet gate) so "best" ranking is recall-aware too -
// see isBetterEra's comment for why that matters.
//
// The threshold matters: a bare ">0" here (the original behavior) let a run converge at
// era 44-46 with the OOS window containing exactly ZERO true Buy/Sell bars that era
// (logged as "OOS recall Buy:n/a Sell:n/a Neutral:100%") - a full Neutral-only collapse
// that the gate waved through because there was nothing to measure recall against, not
// because the model was actually unbiased. Requiring a real minimum sample count means
// an unlucky/thin OOS slice blocks convergence instead of silently passing it.
int buyRecallPct = ( m_oosBuyTotal > = MIN_OOS_CLASS_SAMPLES_FOR_GATE ) ? ( int ) MathRound ( 100.0 * m_oosBuyHits / m_oosBuyTotal ) : -1 ;
int sellRecallPct = ( m_oosSellTotal > = MIN_OOS_CLASS_SAMPLES_FOR_GATE ) ? ( int ) MathRound ( 100.0 * m_oosSellHits / m_oosSellTotal ) : -1 ;
int neutralRecallPct = ( m_oosNeutralTotal > = MIN_OOS_CLASS_SAMPLES_FOR_GATE ) ? ( int ) MathRound ( 100.0 * m_oosNeutralHits / m_oosNeutralTotal ) : -1 ;
logBuyRecallPct = buyRecallPct ;
logSellRecallPct = sellRecallPct ;
logNeutralRecallPct = neutralRecallPct ;
m_lastBuyRecallPct = buyRecallPct ;
m_lastSellRecallPct = sellRecallPct ;
// Predicted-rate (share of ALL OOS bars this era the model called this class, regardless
// of whether that call was right) and precision (of just those calls, how many were
// right) - see logBuyPredPct's declaration comment above for why this is worth logging
// alongside recall. Denominator is the per-era OOS bar count (sum of the per-class true
// totals, all tallied in the same pass-3 block and reset together each era) - NOT
// m_oosSamples, which only resets on a full model reset and so accumulates across every
// era of the run: dividing this era's calls by that all-run total diluted the logged
// rate by roughly the era number (observed: era-15 "Buy:2%" that was really ~30%),
// making a genuinely directional model read as a nearly-dead output.
int oosEraBars = m_oosBuyTotal + m_oosSellTotal + m_oosNeutralTotal ;
logBuyPredPct = ( oosEraBars > 0 ) ? ( int ) MathRound ( 100.0 * m_oosBuyPredicted / oosEraBars ) : -1 ;
logSellPredPct = ( oosEraBars > 0 ) ? ( int ) MathRound ( 100.0 * m_oosSellPredicted / oosEraBars ) : -1 ;
logBuyPrecPct = ( m_oosBuyPredicted > 0 ) ? ( int ) MathRound ( 100.0 * m_oosBuyPredictedHits / m_oosBuyPredicted ) : -1 ;
logSellPrecPct = ( m_oosSellPredicted > 0 ) ? ( int ) MathRound ( 100.0 * m_oosSellPredictedHits / m_oosSellPredicted ) : -1 ;
//--- Live-fired precision (what actually trades - see m_oosBuyFired): of the directional
//--- calls that cleared the confidence floor under the live/prior-corrected rule this era,
//--- how many were right. Cached for the panel/log; -1 = the model fired none this era.
logBuyFiredPrecPct = ( m_oosBuyFired > 0 ) ? ( int ) MathRound ( 100.0 * m_oosBuyFiredHits / m_oosBuyFired ) : -1 ;
logSellFiredPrecPct = ( m_oosSellFired > 0 ) ? ( int ) MathRound ( 100.0 * m_oosSellFiredHits / m_oosSellFired ) : -1 ;
m_lastBuyFiredPrecPct = logBuyFiredPrecPct ;
m_lastSellFiredPrecPct = logSellFiredPrecPct ;
m_lastBuyFired = m_oosBuyFired ;
m_lastSellFired = m_oosSellFired ;
feat(ai): rank checkpoints on directional precision, not balanced accuracy
Balanced accuracy is maximized by exactly the model this system must never
deploy. Measured frontier at fixed signal strength, base rate 6.1%:
tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0%
tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3%
tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5%
It rises monotonically as the model calls MORE and is right LESS, because
two of its three terms are directional recalls that a call-everything model
drives to ~95%, while the Neutral term it sacrifices counts for only a
third. The 2026-07-29 run landed exactly there: balanced 58-64% while
calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base
rate. Only the per-class recall floor stopped those deploying - a guard
doing the job the objective should have been doing - and that same guard
also rejected the genuinely useful sparse-but-precise checkpoints.
Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how
many were right. That is what a trading edge is. Two anti-degenerate floors
bracket it, since precision alone is trivially maximized by calling almost
nothing: coverage must reach a fraction of the true directional base rate
(derived, not configured - it adapts to any symbol/timeframe/label rule),
and precision must at least beat that base rate.
Against the same frontier the deploy order inverts from
tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first)
to
tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage)
Balanced accuracy is kept in the log as a diagnostic and marked as such, so
a run where the two disagree - the signature of an over-caller - is visible
at a glance. MinRecall no longer decides what ships; it now only drives the
diagnostic recall line and is a candidate for removal.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
//--- SELECTION METRIC. Ranking moved off balanced accuracy (macro-recall) 2026-07-30
//--- because that metric is maximized by exactly the model this system must never deploy.
//--- Measured frontier at a fixed signal strength, base rate 6.1%:
//--- tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0%
//--- tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3%
//--- tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5%
//--- Balanced accuracy rises monotonically as the model calls MORE and is right LESS,
//--- because two of its three terms are directional recalls that a call-everything model
//--- drives to ~95% while the Neutral term it sacrifices counts for only a third. The
//--- 2026-07-29 run landed exactly there: balanced 58-64% while calling a direction on
//--- ~100% of bars at a 5-7% win rate against a ~6% base rate - no information at all.
//--- Only the per-class recall floor stopped those from deploying, i.e. a guard was doing
//--- the job the objective should have been doing, and the same guard also rejected the
//--- genuinely useful sparse-but-precise checkpoints (directional recall 4-6%).
//--- Ranking is now DIRECTIONAL PRECISION - of the bars this model called Buy or Sell,
//--- how many were right - which is what a trading edge actually is. Two anti-degenerate
//--- floors bracket it, because precision alone is trivially maximized by calling almost
//--- nothing: coverage must reach a fraction of the true base rate, and precision must at
//--- least beat that base rate (a model no better than the coin is not an edge).
int oosDirCalls = m_oosBuyPredicted + m_oosSellPredicted ;
int oosDirHits = m_oosBuyPredictedHits + m_oosSellPredictedHits ;
int oosDirTrue = m_oosBuyTotal + m_oosSellTotal ;
bool coverageMeasurable = ( oosEraBars > 0 & & oosDirTrue > 0 ) ;
double coveragePct = coverageMeasurable ? 100.0 * oosDirCalls / oosEraBars : -1.0 ;
double baseRatePct = coverageMeasurable ? 100.0 * oosDirTrue / oosEraBars : -1.0 ;
double dirPrecPct = ( oosDirCalls > 0 ) ? 100.0 * oosDirHits / oosDirCalls : -1.0 ;
double minCoveragePct = coverageMeasurable ? baseRatePct * MIN_COVERAGE_FRACTION_OF_BASE_RATE : -1.0 ;
fix: refuse invalid SL/TP, fix the unreachable deploy floor, scale the horizon
Three defects found by reading the 2026-08-01 training logs, all of which
only became visible because the relabel made the numbers mean something.
1. A STALE ENUM TRAINED FOUR MODELS ON THE WRONG TARGET.
`OnInit: trade settings snapshot - SL_Mode=1 TP_Mode=-101`
-101 was TP_PREV_SWING, deleted from TAKE_PROFIT_MODE on 2026-07-31 in
7eb48f5. MetaTrader does not validate a saved enum input against the
enum's current members, so charts saved before that kept the old
integer. BarrierMultiples()'s `if(tpMult <= 0.0) tpMult = slMult;`
then quietly turned it into a 1:1 barrier, and all four topologies
trained ~250 eras against a strategy nobody selected - while the log
reported "target 1.00*ATR" as though it were configured.
Since the relabel these two inputs ARE the label definition, so this
is not a bad trade setting, it is a wrong dataset. ValidateBarrier-
Inputs() now refuses to start (INIT_FAILED + Alert + an explicit fix)
on any value that is not an enum member. Members are enumerated rather
than range-checked because both enums are sparse and carry negative
sentinels, so no min/max test can tell a legal value from a deleted
one - which is the entire failure mode. The fallback survives as
belt-and-braces but now announces itself: a fallback that cannot say
it fired is indistinguishable from correct behaviour.
2. THE DEPLOYABILITY FLOOR BECAME MATHEMATICALLY UNREACHABLE.
`tradeableOK` required `dirPrecPct >= baseRatePct`, where baseRatePct
is Buy+Sell as a share of all bars. At the old exact-pivot target that
was ~6%, so "beat the base rate" read as "beat chance" and the test
looked sound. Triple-barrier labels put it at ~83%, so the gate now
demanded 83% directional precision - impossible by construction.
Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing
safe to deploy" at a perfectly healthy 43-45% precision, with no
checkpoint able to ship however good it got.
Replaced with ZERO-SKILL precision, max(Buy,Sell)/allBars: exactly the
score of the degenerate always-call-one-direction model this floor
exists to reject. Correct at any base rate - ~43% on the current
labels, ~3% on the old rare-pivot ones. The era line now prints
"(chance N%, edge +Mpp)" beside the selection score, because 44%
precision is excellent against a 3% chance level and worthless against
a 43% one, and reading the first as the second is what made tonight's
run look better than it was.
3. THE HORIZON IGNORED THE BARRIER GEOMETRY.
ComputeBarrierHorizonBars() returned the median ZigZag leg, which
measures how long a ~1 ATR move takes and says nothing about how long
the CONFIGURED barrier needs. First-passage time out of [-m,+k] scales
with m*k, so a 1:3 barrier takes ~3x as long as 1:1; the unscaled
horizon would have timed out most 1:3 trades and pushed Neutral
straight back up, re-creating the imbalance the relabel removes.
Now multiplied by slMult*tpMult, calibrated against a real measurement
rather than assumed: the accidental 1:1 run resolved at horizon 12 with
only 16.7% timeouts, so the swing median is the right scale at m*k=1.
Verifiable, not just asserted: the prebuild now counts barriers that
ended on the VERTICAL barrier and reports them as a share of Neutral.
Neutral conflates "timed out" with "stopped out" and only the first
indicts the horizon.
Both builds compile 0 errors / 0 warnings. Forces a retrain - correcting
TP_Mode re-keys the fingerprint (|TB:1:-101 -> |TB:1:3), which is right:
no existing model was trained on the intended target.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:30:49 -04:00
//--- ZERO-SKILL PRECISION: what a model with no information scores on this metric, by
//--- always calling whichever direction is more common. Its precision is that class's
//--- share of ALL bars, because the bars it calls are uncorrelated with the labels.
//--- This REPLACED `dirPrecPct >= baseRatePct` on 2026-08-01, which was wrong the moment
//--- the labels stopped being rare. baseRatePct is Buy+Sell as a share of all bars: at the
//--- old exact-pivot target that was ~6%, so "beat the base rate" read as "beat chance"
//--- and the test looked sound. Triple-barrier labels put it at ~83%, and the gate then
//--- demanded 83% directional precision - unreachable by construction, so NOTHING could
//--- ever deploy. Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing
//--- safe to deploy" at a genuinely healthy 43-45% precision.
//--- max(Buy,Sell) is the right benchmark at ANY base rate: it is exactly the score of the
//--- degenerate always-call-one-direction model this floor exists to reject, and it
//--- degrades correctly to ~3% on the old rare-pivot labels.
double chancePrecPct = coverageMeasurable
? 100.0 * MathMax ( m_oosBuyTotal , m_oosSellTotal ) / oosEraBars : -1.0 ;
feat(ai): rank checkpoints on directional precision, not balanced accuracy
Balanced accuracy is maximized by exactly the model this system must never
deploy. Measured frontier at fixed signal strength, base rate 6.1%:
tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0%
tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3%
tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5%
It rises monotonically as the model calls MORE and is right LESS, because
two of its three terms are directional recalls that a call-everything model
drives to ~95%, while the Neutral term it sacrifices counts for only a
third. The 2026-07-29 run landed exactly there: balanced 58-64% while
calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base
rate. Only the per-class recall floor stopped those deploying - a guard
doing the job the objective should have been doing - and that same guard
also rejected the genuinely useful sparse-but-precise checkpoints.
Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how
many were right. That is what a trading edge is. Two anti-degenerate floors
bracket it, since precision alone is trivially maximized by calling almost
nothing: coverage must reach a fraction of the true directional base rate
(derived, not configured - it adapts to any symbol/timeframe/label rule),
and precision must at least beat that base rate.
Against the same frontier the deploy order inverts from
tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first)
to
tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage)
Balanced accuracy is kept in the log as a diagnostic and marked as such, so
a run where the two disagree - the signature of an over-caller - is visible
at a glance. MinRecall no longer decides what ships; it now only drives the
diagnostic recall line and is a candidate for removal.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
logCoveragePct = ( int ) MathRound ( coveragePct ) ;
logDirPrecPct = ( int ) MathRound ( dirPrecPct ) ;
fix: refuse invalid SL/TP, fix the unreachable deploy floor, scale the horizon
Three defects found by reading the 2026-08-01 training logs, all of which
only became visible because the relabel made the numbers mean something.
1. A STALE ENUM TRAINED FOUR MODELS ON THE WRONG TARGET.
`OnInit: trade settings snapshot - SL_Mode=1 TP_Mode=-101`
-101 was TP_PREV_SWING, deleted from TAKE_PROFIT_MODE on 2026-07-31 in
7eb48f5. MetaTrader does not validate a saved enum input against the
enum's current members, so charts saved before that kept the old
integer. BarrierMultiples()'s `if(tpMult <= 0.0) tpMult = slMult;`
then quietly turned it into a 1:1 barrier, and all four topologies
trained ~250 eras against a strategy nobody selected - while the log
reported "target 1.00*ATR" as though it were configured.
Since the relabel these two inputs ARE the label definition, so this
is not a bad trade setting, it is a wrong dataset. ValidateBarrier-
Inputs() now refuses to start (INIT_FAILED + Alert + an explicit fix)
on any value that is not an enum member. Members are enumerated rather
than range-checked because both enums are sparse and carry negative
sentinels, so no min/max test can tell a legal value from a deleted
one - which is the entire failure mode. The fallback survives as
belt-and-braces but now announces itself: a fallback that cannot say
it fired is indistinguishable from correct behaviour.
2. THE DEPLOYABILITY FLOOR BECAME MATHEMATICALLY UNREACHABLE.
`tradeableOK` required `dirPrecPct >= baseRatePct`, where baseRatePct
is Buy+Sell as a share of all bars. At the old exact-pivot target that
was ~6%, so "beat the base rate" read as "beat chance" and the test
looked sound. Triple-barrier labels put it at ~83%, so the gate now
demanded 83% directional precision - impossible by construction.
Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing
safe to deploy" at a perfectly healthy 43-45% precision, with no
checkpoint able to ship however good it got.
Replaced with ZERO-SKILL precision, max(Buy,Sell)/allBars: exactly the
score of the degenerate always-call-one-direction model this floor
exists to reject. Correct at any base rate - ~43% on the current
labels, ~3% on the old rare-pivot ones. The era line now prints
"(chance N%, edge +Mpp)" beside the selection score, because 44%
precision is excellent against a 3% chance level and worthless against
a 43% one, and reading the first as the second is what made tonight's
run look better than it was.
3. THE HORIZON IGNORED THE BARRIER GEOMETRY.
ComputeBarrierHorizonBars() returned the median ZigZag leg, which
measures how long a ~1 ATR move takes and says nothing about how long
the CONFIGURED barrier needs. First-passage time out of [-m,+k] scales
with m*k, so a 1:3 barrier takes ~3x as long as 1:1; the unscaled
horizon would have timed out most 1:3 trades and pushed Neutral
straight back up, re-creating the imbalance the relabel removes.
Now multiplied by slMult*tpMult, calibrated against a real measurement
rather than assumed: the accidental 1:1 run resolved at horizon 12 with
only 16.7% timeouts, so the swing median is the right scale at m*k=1.
Verifiable, not just asserted: the prebuild now counts barriers that
ended on the VERTICAL barrier and reports them as a share of Neutral.
Neutral conflates "timed out" with "stopped out" and only the first
indicts the horizon.
Both builds compile 0 errors / 0 warnings. Forces a retrain - correcting
TP_Mode re-keys the fingerprint (|TB:1:-101 -> |TB:1:3), which is right:
no existing model was trained on the intended target.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:30:49 -04:00
logChancePrecPct = ( chancePrecPct > = 0.0 ) ? ( int ) MathRound ( chancePrecPct ) : -1 ;
feat(ai): rank checkpoints on directional precision, not balanced accuracy
Balanced accuracy is maximized by exactly the model this system must never
deploy. Measured frontier at fixed signal strength, base rate 6.1%:
tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0%
tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3%
tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5%
It rises monotonically as the model calls MORE and is right LESS, because
two of its three terms are directional recalls that a call-everything model
drives to ~95%, while the Neutral term it sacrifices counts for only a
third. The 2026-07-29 run landed exactly there: balanced 58-64% while
calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base
rate. Only the per-class recall floor stopped those deploying - a guard
doing the job the objective should have been doing - and that same guard
also rejected the genuinely useful sparse-but-precise checkpoints.
Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how
many were right. That is what a trading edge is. Two anti-degenerate floors
bracket it, since precision alone is trivially maximized by calling almost
nothing: coverage must reach a fraction of the true directional base rate
(derived, not configured - it adapts to any symbol/timeframe/label rule),
and precision must at least beat that base rate.
Against the same frontier the deploy order inverts from
tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first)
to
tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage)
Balanced accuracy is kept in the log as a diagnostic and marked as such, so
a run where the two disagree - the signature of an over-caller - is visible
at a glance. MinRecall no longer decides what ships; it now only drives the
diagnostic recall line and is a candidate for removal.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
//--- Deployability. Replaces the per-class recall floor as the gate the checkpoint
//--- selection and the plateau ladder's "is there anything safe to deploy" test read.
//--- MinRecall still drives the diagnostic recall line below, but no longer decides what
//--- ships - it is the input that produced the catch-22 where nothing ever qualified.
fix(autotune): MI scorer read an array nobody filled; add the permutation floor
THE TUNER WAS A SILENT NO-OP. Every chart logged
auto-tune complete - 17 candidate settings scored in ~139s,
feature/label mutual information 0.0000 -> 0.0000 nats (no improvement)
0.0000 is not a weak result, it is a broken measurement: finite-sample MI
is biased UPWARD, so even pure noise scores above zero. Cause:
ScoreCurrentParamsByMI called BufferTempDataCompute(), which APPENDS the
bar's features to TempData and never touches m_featureCache - only the
caching wrapper BufferTempData() writes that array. It then read
m_featureCache, which ReInitADIndicators had just invalidated. Every
column came back constant, FeatureColumnMI returned 0 for all of them,
and all 17 candidates tied at exactly zero. 139 s per chart to return the
settings it started with.
Now reads the values back out of TempData, where they actually land. And
an exactly-zero best score is called out as a fault rather than reported
as "no improvement", because that is what it is.
ADDED: a PERMUTATION BASELINE, which is the diagnostic this project has
been missing. MI's finite-sample bias is ~(bins-1)(classes-1)/(2n) nats -
at these sample sizes the same order as any real edge in this domain - so
a raw MI figure is uninterpretable on its own. Shuffling the labels
destroys every genuine association while leaving sample size, binning and
class proportions intact, so the score it produces IS this dataset's
noise floor, measured rather than approximated. The log now reads
feature/label information - X nats against a shuffled-label floor of Y
and says outright whether the features carry usable information about the
target. It needs no training, no topology and no convergence, so unlike
every accuracy number in this codebase it cannot be confounded by an
optimizer or an objective. If the score sits on the floor, no change of
architecture can help - which is the question the last three days of
zero-edge results have been circling.
DEPLOY FLOOR: `dirPrecPct > chancePrecPct` passed anything above chance by
any amount. At ~11,000 directional calls the standard error of the
precision estimate is ~0.4pp, so that gate was accepting sub-one-sigma
noise - the perceptron deployed at edge +0pp on 2026-08-01. Now requires
EDGE_MIN_SIGMAS (2.0) standard errors above chance, computed from the
actual call count, so the bar scales with the evidence instead of needing
a hand-picked constant.
Recorded with it, because it is why chance is the right reference at all:
under a driftless random walk P(touch +k*ATR before -m*ATR) = m/(m+k),
and the break-even win rate for a k:m reward:risk trade is ALSO m/(m+k).
The label's own base rate IS the break-even rate, at every SL/TP setting.
So "beats chance" and "is profitable" are the same test, and no choice of
SL/TP can manufacture an edge - only prediction can.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:05:50 -04:00
//--- The margin is not arbitrary and not a knob: beating chance by any amount at all is a
//--- coin-flip result once the estimate's own sampling error is accounted for. With
//--- oosDirCalls directional calls at a chance rate p, the standard error of the measured
//--- precision is sqrt(p(1-p)/n) - about 0.4pp at the ~11,000 calls these runs produce - so
//--- `dirPrecPct > chancePrecPct` was passing models whose entire "edge" was under one
//--- sigma. Observed 2026-08-01: the perceptron deployed at edge +0pp.
//--- Requiring EDGE_MIN_SIGMAS standard errors instead scales the bar with the evidence:
//--- a sparse model needs a bigger measured edge to qualify than a dense one, which is
//--- exactly right, and no constant has to be re-tuned when coverage changes.
double chanceP = ( chancePrecPct > = 0.0 ) ? chancePrecPct / 100.0 : 0.0 ;
double precSE = ( oosDirCalls > 0 & & chanceP > 0.0 & & chanceP < 1.0 )
? 100.0 * MathSqrt ( chanceP * ( 1.0 - chanceP ) / oosDirCalls ) : 0.0 ;
double edgeFloorPct = chancePrecPct + EDGE_MIN_SIGMAS * precSE ;
feat(ai): rank checkpoints on directional precision, not balanced accuracy
Balanced accuracy is maximized by exactly the model this system must never
deploy. Measured frontier at fixed signal strength, base rate 6.1%:
tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0%
tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3%
tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5%
It rises monotonically as the model calls MORE and is right LESS, because
two of its three terms are directional recalls that a call-everything model
drives to ~95%, while the Neutral term it sacrifices counts for only a
third. The 2026-07-29 run landed exactly there: balanced 58-64% while
calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base
rate. Only the per-class recall floor stopped those deploying - a guard
doing the job the objective should have been doing - and that same guard
also rejected the genuinely useful sparse-but-precise checkpoints.
Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how
many were right. That is what a trading edge is. Two anti-degenerate floors
bracket it, since precision alone is trivially maximized by calling almost
nothing: coverage must reach a fraction of the true directional base rate
(derived, not configured - it adapts to any symbol/timeframe/label rule),
and precision must at least beat that base rate.
Against the same frontier the deploy order inverts from
tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first)
to
tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage)
Balanced accuracy is kept in the log as a diagnostic and marked as such, so
a run where the two disagree - the signature of an over-caller - is visible
at a glance. MinRecall no longer decides what ships; it now only drives the
diagnostic recall line and is a candidate for removal.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
bool tradeableOK = coverageMeasurable & & dirPrecPct > = 0.0 & &
fix(autotune): MI scorer read an array nobody filled; add the permutation floor
THE TUNER WAS A SILENT NO-OP. Every chart logged
auto-tune complete - 17 candidate settings scored in ~139s,
feature/label mutual information 0.0000 -> 0.0000 nats (no improvement)
0.0000 is not a weak result, it is a broken measurement: finite-sample MI
is biased UPWARD, so even pure noise scores above zero. Cause:
ScoreCurrentParamsByMI called BufferTempDataCompute(), which APPENDS the
bar's features to TempData and never touches m_featureCache - only the
caching wrapper BufferTempData() writes that array. It then read
m_featureCache, which ReInitADIndicators had just invalidated. Every
column came back constant, FeatureColumnMI returned 0 for all of them,
and all 17 candidates tied at exactly zero. 139 s per chart to return the
settings it started with.
Now reads the values back out of TempData, where they actually land. And
an exactly-zero best score is called out as a fault rather than reported
as "no improvement", because that is what it is.
ADDED: a PERMUTATION BASELINE, which is the diagnostic this project has
been missing. MI's finite-sample bias is ~(bins-1)(classes-1)/(2n) nats -
at these sample sizes the same order as any real edge in this domain - so
a raw MI figure is uninterpretable on its own. Shuffling the labels
destroys every genuine association while leaving sample size, binning and
class proportions intact, so the score it produces IS this dataset's
noise floor, measured rather than approximated. The log now reads
feature/label information - X nats against a shuffled-label floor of Y
and says outright whether the features carry usable information about the
target. It needs no training, no topology and no convergence, so unlike
every accuracy number in this codebase it cannot be confounded by an
optimizer or an objective. If the score sits on the floor, no change of
architecture can help - which is the question the last three days of
zero-edge results have been circling.
DEPLOY FLOOR: `dirPrecPct > chancePrecPct` passed anything above chance by
any amount. At ~11,000 directional calls the standard error of the
precision estimate is ~0.4pp, so that gate was accepting sub-one-sigma
noise - the perceptron deployed at edge +0pp on 2026-08-01. Now requires
EDGE_MIN_SIGMAS (2.0) standard errors above chance, computed from the
actual call count, so the bar scales with the evidence instead of needing
a hand-picked constant.
Recorded with it, because it is why chance is the right reference at all:
under a driftless random walk P(touch +k*ATR before -m*ATR) = m/(m+k),
and the break-even win rate for a k:m reward:risk trade is ALSO m/(m+k).
The label's own base rate IS the break-even rate, at every SL/TP setting.
So "beats chance" and "is profitable" are the same test, and no choice of
SL/TP can manufacture an edge - only prediction can.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:05:50 -04:00
coveragePct > = minCoveragePct & & dirPrecPct > edgeFloorPct ;
2026-07-30 10:40:36 -04:00
//--- Ranking key: precision, DISCOUNTED by how far short of the coverage floor the era
//--- fell. Raw precision was wrong here and the 2026-07-30 run caught it within 8 eras -
//--- HYBRID made exactly ONE directional call, got it right, scored 100%, and locked that
//--- in as best-ever. Nothing can beat 100%, so the checkpoint was frozen on a single
//--- sample and the run could only burn to the era cap. The coverage floor was already
//--- computed and already blocked that era from being DEPLOYABLE, but the ranking ignored
//--- it whenever no era had qualified yet - which is exactly the phase this matters in.
//--- Discounting rather than thresholding keeps the ordering continuous: an era at half
//--- the floor scores half its precision, so more coverage and better precision both
//--- improve rank and neither can be traded away entirely. Above the floor the credit
//--- saturates at 1.0, so ranking among genuinely deployable eras stays pure precision.
double coverageCredit = 1.0 ;
if ( minCoveragePct > 0.0 & & coveragePct > = 0.0 )
coverageCredit = MathMin ( 1.0 , coveragePct / minCoveragePct ) ;
double selectionScore = ( dirPrecPct > = 0.0 ) ? dirPrecPct * coverageCredit : 0.0 ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
bool directionalRecallOK = ( buyRecallPct < 0 | | buyRecallPct > = m_minDirectionalRecallPct ) & &
( sellRecallPct < 0 | | sellRecallPct > = m_minDirectionalRecallPct ) & &
( neutralRecallPct < 0 | | neutralRecallPct > = m_minDirectionalRecallPct ) ;
// Balanced accuracy (macro-recall): the mean of the three per-class recalls - the metric
// the checkpoint SELECTION ranks on (see m_bestBalancedOos). Unlike blended accuracy it
// weights Buy, Sell and Neutral equally, so it can't be inflated by the ~96%-Neutral base
// rate. Computed from the same raw per-era recalls the floor uses (not smoothed - the
// whole recall-driven side of this block is per-era-raw by design). When a directional
// class is thin/unmeasured this era (recall -1), balanced accuracy isn't meaningful, so
// fall back to the blended dOosForecast for ranking that era (prior behavior) rather than
// averaging a partial set - the recall floor + directionalRecallMeasured still guard the
// actual convergence decision separately.
double balancedOosEra = ( buyRecallPct > = 0 & & sellRecallPct > = 0 & & neutralRecallPct > = 0 )
? ( buyRecallPct + sellRecallPct + neutralRecallPct ) / 3.0
: dOosForecast ;
logBalancedAccPct = ( buyRecallPct > = 0 & & sellRecallPct > = 0 & & neutralRecallPct > = 0 )
? ( int ) MathRound ( balancedOosEra ) : -1 ;
// A real (non-thin-sample, i.e. not the -1 "n/a" sentinel) 0% recall on any class means
// the model never once got that class right this era - a majority-class collapse
// (predict-everything-Neutral, or symmetrically a Buy/Sell-only collapse), not progress
// toward separating classes. Before any era has ever passed the recall floor,
// isBetterEra's fallback below is a pure blended-accuracy tiebreak, and blended accuracy
// is trivially maximized by collapsing to the majority class. Observed in practice
// (2026-07-19, SP500 H4): once a run landed on a 0%/0%/100% Buy/Sell/Neutral era, its
// accuracy kept creeping upward for 124 STRAIGHT eras purely from sharpening the
// Neutral-vs-everything boundary - each tick registered as a "new best", re-anchoring the
// checkpoint AND bumping eta back toward its ceiling (the recovery bump below), actively
// rewarding the collapse instead of remaining neutral to it. Excluding these eras from
// isBetterEra denies them that anchor/reward without touching the restore/decay branch
// below, which stays exactly as gated on m_bestPassedRecall as before - see that block's
// own comment for why loosening THAT part pre-pass caused a worse failure historically.
feat(ai): rank checkpoints on directional precision, not balanced accuracy
Balanced accuracy is maximized by exactly the model this system must never
deploy. Measured frontier at fixed signal strength, base rate 6.1%:
tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0%
tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3%
tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5%
It rises monotonically as the model calls MORE and is right LESS, because
two of its three terms are directional recalls that a call-everything model
drives to ~95%, while the Neutral term it sacrifices counts for only a
third. The 2026-07-29 run landed exactly there: balanced 58-64% while
calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base
rate. Only the per-class recall floor stopped those deploying - a guard
doing the job the objective should have been doing - and that same guard
also rejected the genuinely useful sparse-but-precise checkpoints.
Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how
many were right. That is what a trading edge is. Two anti-degenerate floors
bracket it, since precision alone is trivially maximized by calling almost
nothing: coverage must reach a fraction of the true directional base rate
(derived, not configured - it adapts to any symbol/timeframe/label rule),
and precision must at least beat that base rate.
Against the same frontier the deploy order inverts from
tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first)
to
tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage)
Balanced accuracy is kept in the log as a diagnostic and marked as such, so
a run where the two disagree - the signature of an over-caller - is visible
at a glance. MinRecall no longer decides what ships; it now only drives the
diagnostic recall line and is a candidate for removal.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
//--- Under precision ranking the degenerate era is the one that called NOTHING
//--- directional (precision undefined, nothing to trade), not one whose per-class
//--- recall touched zero - a sparse high-precision model legitimately has low recall.
bool isFullyCollapsedEra = ( oosDirCalls < = 0 ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// Lexicographic "better than the best-so-far" ordering: passing the directional recall
// floor always outranks not passing it, regardless of blended dOosForecast; only WITHIN
// the same pass/fail category does blended accuracy break the tie. Without this, an era
// that traded a few "safe" Neutral calls for genuinely useful (recall-improving) Buy/Sell
// calls would look like a regression in blended-accuracy-only terms and get its
// checkpoint skipped / learning rate cut - fighting directly against the network learning
// to call Buy/Sell at all, since Neutral is the large majority class (~80%+ of labels) and
// a model that just calls everything Neutral already scores well on blended accuracy
// alone. isWorseEra mirrors the same ordering for the eta-decay-on-regression trigger.
// (directionalRecallOK implies !isFullyCollapsedEra already, since the floor is always
// >0%, so the first clause below needs no extra guard - only the pre-pass accuracy-only
// tiebreak in the second clause does.)
// Within the same recall-pass category the tie now breaks on BALANCED accuracy, not the
// Neutral-dominated blended dOosForecast - see m_bestBalancedOos. This is what deploys the
// most class-balanced era instead of the most Neutral-leaning one, and it also strengthens
// the pre-pass phase: a Neutral-only era scores (0+0+N)/3 in balanced terms (low) rather
// than the ~80% it scores in blended terms, so it can no longer re-anchor the checkpoint.
feat(ai): rank checkpoints on directional precision, not balanced accuracy
Balanced accuracy is maximized by exactly the model this system must never
deploy. Measured frontier at fixed signal strength, base rate 6.1%:
tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0%
tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3%
tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5%
It rises monotonically as the model calls MORE and is right LESS, because
two of its three terms are directional recalls that a call-everything model
drives to ~95%, while the Neutral term it sacrifices counts for only a
third. The 2026-07-29 run landed exactly there: balanced 58-64% while
calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base
rate. Only the per-class recall floor stopped those deploying - a guard
doing the job the objective should have been doing - and that same guard
also rejected the genuinely useful sparse-but-precise checkpoints.
Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how
many were right. That is what a trading edge is. Two anti-degenerate floors
bracket it, since precision alone is trivially maximized by calling almost
nothing: coverage must reach a fraction of the true directional base rate
(derived, not configured - it adapts to any symbol/timeframe/label rule),
and precision must at least beat that base rate.
Against the same frontier the deploy order inverts from
tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first)
to
tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage)
Balanced accuracy is kept in the log as a diagnostic and marked as such, so
a run where the two disagree - the signature of an over-caller - is visible
at a glance. MinRecall no longer decides what ships; it now only drives the
diagnostic recall line and is a candidate for removal.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
//--- tradeableOK / selectionScore, not directionalRecallOK / balancedOosEra - see the
//--- SELECTION METRIC note above. The lexicographic shape is unchanged: qualifying
//--- always outranks not qualifying, and the score only breaks ties within a category.
bool isBetterEra = ( tradeableOK & & ! m_bestPassedRecall ) | |
( tradeableOK = = m_bestPassedRecall & & ! isFullyCollapsedEra & & selectionScore > m_bestBalancedOos ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// The recall-pass-loss clause used to fire on ANY drop out of a full 3-way recall pass,
// even a near-miss on one class at unchanged accuracy (e.g. observed: Buy:56% Sell:41%
// Neutral:34% - Neutral alone missing the 40% floor by a few points) - treating that
// identically to a total collapse back to Neutral-only. With three classes all needing
// to simultaneously clear the floor, that made isWorseEra fire on most eras once a pass
// was ever achieved, ratcheting eta toward ETA_MIN within a handful of eras and then
// (before the recovery bump below existed) leaving it stuck there permanently - visible
// in practice as ~25 back-to-back identical "regressed from best 70.6% to 70.6%" eras.
// Now only counts as worse if accuracy ALSO dropped meaningfully (same threshold
// regardless of whether the recall-pass flag changed too) - losing the recall-pass flag
// at flat/improved accuracy is borderline variance, not a regression worth
// restoring+decaying over.
feat(ai): rank checkpoints on directional precision, not balanced accuracy
Balanced accuracy is maximized by exactly the model this system must never
deploy. Measured frontier at fixed signal strength, base rate 6.1%:
tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0%
tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3%
tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5%
It rises monotonically as the model calls MORE and is right LESS, because
two of its three terms are directional recalls that a call-everything model
drives to ~95%, while the Neutral term it sacrifices counts for only a
third. The 2026-07-29 run landed exactly there: balanced 58-64% while
calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base
rate. Only the per-class recall floor stopped those deploying - a guard
doing the job the objective should have been doing - and that same guard
also rejected the genuinely useful sparse-but-precise checkpoints.
Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how
many were right. That is what a trading edge is. Two anti-degenerate floors
bracket it, since precision alone is trivially maximized by calling almost
nothing: coverage must reach a fraction of the true directional base rate
(derived, not configured - it adapts to any symbol/timeframe/label rule),
and precision must at least beat that base rate.
Against the same frontier the deploy order inverts from
tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first)
to
tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage)
Balanced accuracy is kept in the log as a diagnostic and marked as such, so
a run where the two disagree - the signature of an over-caller - is visible
at a glance. MinRecall no longer decides what ships; it now only drives the
diagnostic recall line and is a candidate for removal.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
bool isWorseEra = selectionScore < m_bestBalancedOos - ETA_DECAY_REGRESSION_PCT ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
if ( isBetterEra )
{
//--- Snapshot BOTH scores at the checkpoint: m_bestBalancedOos is what ranking compares
//--- against next era; m_bestOosForecast keeps the blended value FinalizeTrainRun() and
//--- the restore branch reset dOosForecast to (see m_bestBalancedOos' declaration).
m_bestOosForecast = dOosForecast ;
feat(ai): rank checkpoints on directional precision, not balanced accuracy
Balanced accuracy is maximized by exactly the model this system must never
deploy. Measured frontier at fixed signal strength, base rate 6.1%:
tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0%
tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3%
tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5%
It rises monotonically as the model calls MORE and is right LESS, because
two of its three terms are directional recalls that a call-everything model
drives to ~95%, while the Neutral term it sacrifices counts for only a
third. The 2026-07-29 run landed exactly there: balanced 58-64% while
calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base
rate. Only the per-class recall floor stopped those deploying - a guard
doing the job the objective should have been doing - and that same guard
also rejected the genuinely useful sparse-but-precise checkpoints.
Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how
many were right. That is what a trading edge is. Two anti-degenerate floors
bracket it, since precision alone is trivially maximized by calling almost
nothing: coverage must reach a fraction of the true directional base rate
(derived, not configured - it adapts to any symbol/timeframe/label rule),
and precision must at least beat that base rate.
Against the same frontier the deploy order inverts from
tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first)
to
tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage)
Balanced accuracy is kept in the log as a diagnostic and marked as such, so
a run where the two disagree - the signature of an over-caller - is visible
at a glance. MinRecall no longer decides what ships; it now only drives the
diagnostic recall line and is a candidate for removal.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
m_bestBalancedOos = selectionScore ;
m_bestPassedRecall = tradeableOK ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- eval candidates are throwaway - track the score (above) but never write a checkpoint
//--- file; m_haveOosCheckpoint=false then also skips the worse-era RestoreWeights() restore.
//--- In-MEMORY weight snapshot (not a file): the file-based checkpoint re-created every
//--- neuron on restore, which the CPU-DLL backend can't do for a second live set - see
//--- CNet::CaptureWeights/RestoreWeights. eval candidates snapshot nothing.
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
m_haveOosCheckpoint = Net . CaptureWeights ( ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// Recovery bump: ETA_DECAY_FACTOR-only ever shrinks eta, and previously nothing ever
// grew it back - a losing streak early in a run (even a since-corrected one) would
// permanently cap how fast every later era could learn for the rest of the run, all
// the way down to ETA_MIN with no way back. A genuinely better era (new best, not
// just a tie) means the current eta is working, so nudge it back up a bit - capped at
// this model's own configured starting rate (m_etaCeiling - AdamLearningRate for
// ADAM, SgdLearningRate for SGD, see that member's declaration comment) so this
// can't runaway past the rate training was actually tuned to start at.
eta = MathMin ( m_etaCeiling , eta / ETA_DECAY_FACTOR ) ;
}
else
if ( isWorseEra & & m_bestOosForecast > 0 )
{
// Decaying eta alone only softens FUTURE steps - it does nothing to undo the
// regression this era already baked into the weights, so a run could (and in
// practice did) spend 15+ eras compounding forward from one bad era's damage,
// each new era fighting the last one's overshoot instead of building on the best
// state found so far. Restore the last checkpointed-good weights before continuing
// (mirrors what FinalizeTrainRun() does at the END of a run, just applied live so
// the oscillation can't compound within a single run) - this is what actually turns
// "reduce LR on regression" into "step back, then retry slower", not just "drift
// slower".
//
// BOTH the restore AND the eta decay below are gated on m_bestPassedRecall: before
// ANY era has ever cleared the per-class recall floor, isBetterEra's own
// lexicographic ordering degrades to a pure blended-accuracy tiebreak
// (directionalRecallOK==false on both sides of the comparison), so "best checkpoint"
// during that phase just means "called Neutral most confidently so far" - restoring
// it would actively defend the majority-class collapse against any era that trades
// some accuracy for real Buy/Sell recall, which is exactly the bias this whole
// recall-gate mechanism exists to prevent (see isBetterEra's own comment above).
// Observed in practice: era 1-3 all "improved" on accuracy alone
// (24.9%->41.4%->52.3%) while Buy/Sell recall stayed at a flat 0% the entire time -
// restoring pre-pass would have locked training into that trajectory instead of
// letting it explore past it. Decaying eta has the same bias one step removed:
// every regression relative to a Neutral-collapse "best" shrinks eta a little more,
// steadily strangling the exploration needed to escape that collapse until eta
// bottoms out at ETA_MIN with no real solution ever found and no checkpoint to fall
// back on either - observed in practice as a run whose best-ever blended accuracy
// kept landing on 0%/0%/100% Buy/Sell/Neutral recall eras, each one triggering
// another decay on the very next era, until eta floored out around era 20 and the
// remaining eras just oscillated between collapse states with no way to make a
// large-enough move to escape and no way to reset. Once m_bestPassedRecall is true,
// there IS a genuinely good state worth protecting, and both restoring the
// checkpoint and decaying eta on regression are safe/correct again.
fix(training): escape the recall-gate catch-22 that let runs decay unchecked
Evidence (MQL5\Logs, SP500 H1, 2026-07-29):
Perceptron era 61 Buy 32% Sell 27% Neut 94% bal 51%
LSTM era 160 Buy 16% Sell 11% Neut 98% bal 42% (peaked 49% @ era 44)
Hybrid era 179 Buy 5% Sell 2% Neut 99% bal 35% (peaked 41%)
CONV era 228 Buy 2% Sell 4% Neut 99% bal 35% (peaked 40% @ era 122)
Every model peaks early then decays monotonically toward Neutral, and nothing
stops it: the restore-best-weights + decay-eta handler is gated on
m_bestPassedRecall, which stays false forever when no checkpoint ever clears the
per-class floor. CONV ran 228 eras with eta pinned at its 0.000300 start. The
plateau ladder cannot end such a run either (stage 3 refuses to deploy without a
recall pass, so it resets ~27 times), making it a 1000-era one-way trip.
The gate's own justification had expired. It was written when the pre-pass
tiebreak was blended-accuracy-only, where "best" really did mean "called Neutral
most confidently". The balanced-selection change replaced that with
`balancedOosEra > m_bestBalancedOos` plus an isFullyCollapsedEra exclusion, so a
Neutral-only era now scores ~33% - the FLOOR of the balanced metric - and cannot
anchor the checkpoint at all. Pre-pass "best" now means "most class-balanced so
far", which is worth defending; and isWorseEra is itself a balanced-accuracy
regression, so it cannot fire merely for trading Neutral calls for Buy/Sell.
The original concern still holds while the best-so-far IS near-collapse, so the
escape is margin-guarded: defend the checkpoint only once balanced accuracy sits
more than BALANCED_WORTH_DEFENDING_MARGIN_PCT (5pp) above the one-class floor of
100/3. Against the run above that engages for all three stuck topologies
(42.3/41.3/50.0 vs a 38.3 threshold) while a genuinely collapsed run still
explores freely.
Two inputs restored to the regime that actually produced a deploy:
- MinRecall 60 -> 40. The one successful auto-deploy in the logs (Hybrid, 28th
00:50, best balanced 66.0%) ran against a 40% floor. 60 has never been shown
reachable here - a floor above what the config can reach is the same "target
set too high" failure the surrounding comment already warns about.
- OversampleParity 60 -> 90. 60 overcorrected. Runs now START Neutral-dominant
(Buy 0-11% recall at era 1) and call Buy/Sell on 0-4% of bars against a ~6%
true base rate - under-calling, with no headroom to converge down from. The
deploying run began at Buy 90% / Sell 36%, 24% of bars called, and settled into
the floor from above. Raw over-calling is the intended starting condition; live
calls are base-rate-calibrated by AILogitPriorStrength, which is why the input's
own note says to judge over-calling by live-fired precision, not raw counts.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 07:51:08 -04:00
// 2026-07-29: the m_bestPassedRecall gate above has an escape now, because its
// stated premise expired. It was written when the pre-pass tiebreak really was
// blended-accuracy-only; the balanced-selection change (m_bestBalancedOos) replaced
// that with `balancedOosEra > m_bestBalancedOos` AND an isFullyCollapsedEra
// exclusion, so a Neutral-only era now scores ~33% (the FLOOR of the balanced
// metric) and cannot anchor the checkpoint at all. "Best checkpoint" pre-pass
// therefore no longer means "called Neutral most confidently" - it means "most
// class-balanced state found so far", which is worth defending, and isWorseEra is
// itself a balanced-accuracy regression, so it cannot fire merely for trading
// Neutral calls for Buy/Sell.
//
// Leaving the gate absolute had a failure mode of its own, and it is not
// hypothetical: if NO checkpoint ever clears the recall floor, m_bestPassedRecall
// stays false forever, so there is never any restore and never any eta decay.
// Observed on SP500 H1 2026-07-29 across three topologies - CONV ran 228 eras with
// eta pinned at its 0.000300 start while balanced accuracy slid 40% -> 35% and Buy
// recall 11% -> 2%. The run had no regression control whatsoever, and the plateau
// ladder could not end it either (stage 3 refuses to deploy without a recall pass),
// so it was a 1000-era one-way trip into a Neutral collapse.
//
// The original concern still applies while the best-so-far IS near-collapse:
// decaying eta against such a "best" strangles the exploration needed to escape it.
// So the escape is margin-guarded - defend the checkpoint only once it sits clearly
// above the one-class floor, which is exactly when there is something real to lose.
bool bestWorthDefending = ( m_bestBalancedOos >
BALANCED_COLLAPSE_PCT + BALANCED_WORTH_DEFENDING_MARGIN_PCT ) ;
if ( m_bestPassedRecall | | bestWorthDefending )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
if ( m_haveOosCheckpoint & & Net . RestoreWeights ( ) )
dOosForecast = m_bestOosForecast ;
if ( eta > ETA_MIN )
eta = MathMax ( ETA_MIN , eta * ETA_DECAY_FACTOR ) ;
feat(ai): measure precision per confidence tier; fix stale metric labels
Two things the 2026-07-30 run exposed.
1. Every user-facing message still called the selection metric "balanced
accuracy". It has ranked on directional precision since a142749, so
"CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5%
PRECISION as if it were macro-recall, while the same era logged an
actual balanced accuracy of 49%. Two different numbers under one
name, in the line that announces a deploy. Relabelled at every site,
including the stage-3 refusal, which still described the per-class
recall floor that stopped being the gate.
2. Precision is now bucketed by confidence tier and logged per era,
both per-tier and cumulatively from each tier upward:
| tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ...
The per-tier number says whether confidence is calibrated to
correctness at all; if it does not rise T0->T3, raising the floor
buys nothing and that is the finding. The ">=" number is what a floor
would actually deliver, with its fire count, so the coverage cost is
visible in the same line. Tier weights are 25/50/75/100, so for an
AI-only config Min_Vote_Open maps straight across: 50 = ">=T1",
75 = ">=T2", 100 = ">=T3".
Bucketing happens at the existing live-fired accounting site, so it
measures exactly the population that trades - not the raw argmax.
Both builds compile 0 errors, 0 warnings. No retrain needed for either.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
Print ( ID + " : OOS selection score (coverage-weighted dir-precision) regressed from best " + DoubleToString ( m_bestBalancedOos , 1 ) +
2026-07-30 15:20:30 -04:00
" % to " + DoubleToString ( selectionScore , 1 ) + " % (blended " + DoubleToString ( m_bestOosForecast , 1 ) +
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
" %-> " + DoubleToString ( dOosForecast , 1 ) + " %) - restoring best checkpoint and decaying learning rate to " + DoubleToString ( eta , 6 ) ) ;
}
else
feat(ai): measure precision per confidence tier; fix stale metric labels
Two things the 2026-07-30 run exposed.
1. Every user-facing message still called the selection metric "balanced
accuracy". It has ranked on directional precision since a142749, so
"CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5%
PRECISION as if it were macro-recall, while the same era logged an
actual balanced accuracy of 49%. Two different numbers under one
name, in the line that announces a deploy. Relabelled at every site,
including the stage-3 refusal, which still described the per-class
recall floor that stopped being the gate.
2. Precision is now bucketed by confidence tier and logged per era,
both per-tier and cumulatively from each tier upward:
| tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ...
The per-tier number says whether confidence is calibrated to
correctness at all; if it does not rise T0->T3, raising the floor
buys nothing and that is the finding. The ">=" number is what a floor
would actually deliver, with its fire count, so the coverage cost is
visible in the same line. Tier weights are 25/50/75/100, so for an
AI-only config Min_Vote_Open maps straight across: 50 = ">=T1",
75 = ">=T2", 100 = ">=T3".
Bucketing happens at the existing live-fired accounting site, so it
measures exactly the population that trades - not the raw argmax.
Both builds compile 0 errors, 0 warnings. No retrain needed for either.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
Print ( ID + " : OOS selection score (coverage-weighted dir-precision) regressed from best " + DoubleToString ( m_bestBalancedOos , 1 ) +
2026-07-30 15:20:30 -04:00
" % to " + DoubleToString ( selectionScore , 1 ) + " % (blended " + DoubleToString ( m_bestOosForecast , 1 ) +
fix(training): escape the recall-gate catch-22 that let runs decay unchecked
Evidence (MQL5\Logs, SP500 H1, 2026-07-29):
Perceptron era 61 Buy 32% Sell 27% Neut 94% bal 51%
LSTM era 160 Buy 16% Sell 11% Neut 98% bal 42% (peaked 49% @ era 44)
Hybrid era 179 Buy 5% Sell 2% Neut 99% bal 35% (peaked 41%)
CONV era 228 Buy 2% Sell 4% Neut 99% bal 35% (peaked 40% @ era 122)
Every model peaks early then decays monotonically toward Neutral, and nothing
stops it: the restore-best-weights + decay-eta handler is gated on
m_bestPassedRecall, which stays false forever when no checkpoint ever clears the
per-class floor. CONV ran 228 eras with eta pinned at its 0.000300 start. The
plateau ladder cannot end such a run either (stage 3 refuses to deploy without a
recall pass, so it resets ~27 times), making it a 1000-era one-way trip.
The gate's own justification had expired. It was written when the pre-pass
tiebreak was blended-accuracy-only, where "best" really did mean "called Neutral
most confidently". The balanced-selection change replaced that with
`balancedOosEra > m_bestBalancedOos` plus an isFullyCollapsedEra exclusion, so a
Neutral-only era now scores ~33% - the FLOOR of the balanced metric - and cannot
anchor the checkpoint at all. Pre-pass "best" now means "most class-balanced so
far", which is worth defending; and isWorseEra is itself a balanced-accuracy
regression, so it cannot fire merely for trading Neutral calls for Buy/Sell.
The original concern still holds while the best-so-far IS near-collapse, so the
escape is margin-guarded: defend the checkpoint only once balanced accuracy sits
more than BALANCED_WORTH_DEFENDING_MARGIN_PCT (5pp) above the one-class floor of
100/3. Against the run above that engages for all three stuck topologies
(42.3/41.3/50.0 vs a 38.3 threshold) while a genuinely collapsed run still
explores freely.
Two inputs restored to the regime that actually produced a deploy:
- MinRecall 60 -> 40. The one successful auto-deploy in the logs (Hybrid, 28th
00:50, best balanced 66.0%) ran against a 40% floor. 60 has never been shown
reachable here - a floor above what the config can reach is the same "target
set too high" failure the surrounding comment already warns about.
- OversampleParity 60 -> 90. 60 overcorrected. Runs now START Neutral-dominant
(Buy 0-11% recall at era 1) and call Buy/Sell on 0-4% of bars against a ~6%
true base rate - under-calling, with no headroom to converge down from. The
deploying run began at Buy 90% / Sell 36%, 24% of bars called, and settled into
the floor from above. Raw over-calling is the intended starting condition; live
calls are base-rate-calibrated by AILogitPriorStrength, which is why the input's
own note says to judge over-calling by live-fired precision, not raw counts.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 07:51:08 -04:00
" %-> " + DoubleToString ( dOosForecast , 1 ) + " %) - best so far is still within " +
DoubleToString ( BALANCED_WORTH_DEFENDING_MARGIN_PCT , 1 ) + " pp of the " +
DoubleToString ( BALANCED_COLLAPSE_PCT , 1 ) + " % one-class floor, so there is nothing worth " +
" restoring yet - continuing to explore without decaying eta (still " + DoubleToString ( eta , 6 ) + " ) " ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
//=== PLATEAU LADDER ====================================================================
//--- Neither branch above fires in the dead zone between "new best" and "regressed by more
//--- than ETA_DECAY_REGRESSION_PCT". This is the response to sitting in it: count eras since
//--- the last new best and escalate. See the PLATEAU_* constants for the full rationale and
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
if ( isBetterEra )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- Moving again: retire the ladder. Deliberately does NOT undo the warm restart's
//--- raised eta - if the bigger step is what produced this new best, decaying it back
//--- would undo the very change that worked. The normal per-era eta schedule takes
//--- over from here.
if ( m_plateauStage > 0 )
Print ( ID + " : new best selection score (coverage-weighted dir-precision) " + DoubleToString ( m_bestBalancedOos , 1 ) +
" % - plateau escape worked, clearing plateau stage " + IntegerToString ( m_plateauStage ) ) ;
m_erasSinceBestBalanced = 0 ;
m_plateauStage = 0 ;
}
else
{
m_erasSinceBestBalanced + + ;
int dueStage = m_erasSinceBestBalanced / PLATEAU_PATIENCE_ERAS ;
if ( dueStage > m_plateauStage )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
m_plateauStage = dueStage ;
string stageNote = IntegerToString ( m_erasSinceBestBalanced ) + " eras with no new best selection score (best " +
DoubleToString ( m_bestBalancedOos , 1 ) + " %) " ;
if ( m_plateauStage = = PLATEAU_STAGE_RESTART | | m_plateauStage = = PLATEAU_STAGE_ANNEAL )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- WARM RESTART: jump eta back to this model's configured starting rate. A
//--- plateau needs a bigger step to climb out of its basin, not a smaller one.
double etaBefore = eta ;
eta = m_etaCeiling ;
//--- The focal-gamma anneal that used to accompany this went with focal loss
//--- on 2026-07-31. It was only ever a monotone step toward zero on a second
//--- imbalance correction; the warm restart above is and always was the
//--- actual escape, so both ladder stages keep their distinct patience
//--- thresholds and simply retry the restart.
Print ( ID + " : PLATEAU stage " + IntegerToString ( m_plateauStage ) + " - " + stageNote +
" . Warm restart: learning rate " + DoubleToString ( etaBefore , 6 ) + " -> " + DoubleToString ( eta , 6 ) +
" . Best checkpoint is safe - this only changes how the NEXT eras train. " ) ;
}
else
if ( m_plateauStage > = PLATEAU_STAGE_DEPLOY )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- Exhausted: both escapes were tried and neither found a better model, so
//--- this IS the best this configuration reaches. The deploy itself happens in
//--- the era-cap/plateau branch at the TOP of the next era, which reuses the
//--- proven "stop + mark complete -> FinalizeTrainRun restores and deploys the
//--- best checkpoint" path rather than duplicating it here.
//--- Safety: only ever auto-deploys a checkpoint that CLEARED the per-class
//--- recall floor (m_bestPassedRecall). If nothing ever did, there is no model
//--- worth deploying - so the ladder resets and keeps trying instead, leaving
//--- the era cap as the ultimate backstop. That is what stops "train to the
//--- best possible result" from degenerating into "deploy a Neutral collapse".
if ( m_bestPassedRecall & & m_haveOosCheckpoint )
Print ( ID + " : PLATEAU stage " + IntegerToString ( PLATEAU_STAGE_DEPLOY ) + " - " + stageNote +
" across " + IntegerToString ( PLATEAU_STAGE_DEPLOY - 1 ) + " warm restarts. Training has converged on what this "
+ " configuration can reach - deploying the best checkpoint (dir-precision "
+ DoubleToString ( m_bestBalancedOos , 1 ) + " %, blended " + DoubleToString ( m_bestOosForecast , 1 ) + " %). " ) ;
else
{
Print ( ID + " : PLATEAU stage " + IntegerToString ( PLATEAU_STAGE_DEPLOY ) + " - " + stageNote +
" , but no checkpoint has ever cleared the deployability floor (directional calls on " +
" at least a quarter as many bars as actually swing, at a precision above that base rate), so there is nothing safe to "
+ " deploy. Restarting the plateau ladder and continuing to train rather than deploying a "
+ " one-class model; the " + IntegerToString ( m_maxErasPerRun ) + " -era cap remains the backstop. " ) ;
m_erasSinceBestBalanced = 0 ;
m_plateauStage = 0 ;
}
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
}
}
m_oosWindow . Add ( dOosForecast ) ;
while ( m_oosWindow . Total ( ) > STABILITY_WINDOW )
m_oosWindow . Delete ( 0 ) ;
m_oosStable = false ;
if ( m_oosWindow . Total ( ) > = STABILITY_WINDOW )
{
double oosMin = m_oosWindow . At ( 0 ) , oosMax = m_oosWindow . At ( 0 ) ;
for ( int w = 1 ; w < m_oosWindow . Total ( ) ; w + + )
{
oosMin = MathMin ( oosMin , m_oosWindow . At ( w ) ) ;
oosMax = MathMax ( oosMax , m_oosWindow . At ( w ) ) ;
}
m_oosStable = ( oosMax - oosMin ) < = STABILITY_TOLERANCE ;
}
// The dError<0.1 RMS-error floor is meaningful for the single-neuron regression head
// (m_outputNeuronsCount==1), where it's the only convergence signal available. For the
// 3-neuron one-hot classification head it's redundant with, and far stricter than,
// dOosForecast/directionalRecallOK: reaching RMS error 0.1 across 3 one-hot targets
// needs every output neuron within ~0.17 of its target on average, i.e. near-perfect
// confident calibration on EVERY bar, not just correct argmax calls - unreachable in
// practice under normal market label noise, so classification runs would oscillate
// forever (era after era hitting good OOS accuracy and passing recall, but never
// satisfying this) without this carve-out.
bool errorGateOK = ( m_outputNeuronsCount = = 3 ) ? true : ( dError < 0.1 ) ;
// Convergence (unlike isBetterEra's ranking) FINALIZES the model, so both directional
// classes must have actually been MEASURED this era. An n/a (-1, thin-sample) Buy or
// Sell recall passing directionalRecallOK is deliberate for ranking (early thin
// windows shouldn't deadlock "best" tracking), but letting it pass HERE converges on
// a window that contained no directional bars to disprove the model. Observed
// 2026-07-19: a mid-run label-cache wipe relabeled the whole window Neutral,
// "accuracy" hit 84.9% with Buy/Sell recall both n/a - without this gate a Neutral-only
// model finalizes as a certified success.
bool directionalRecallMeasured = ( m_outputNeuronsCount ! = 3 ) | | ( buyRecallPct > = 0 & & sellRecallPct > = 0 ) ;
//--- VALIDITY of this era's model, no longer "did it hit a target accuracy". The absolute
//--- OOS-accuracy target (the old MinWR input) is gone: an accuracy number typed in ahead of
//--- time is either unreachable for the symbol/timeframe - in which case the run never
//--- converges and burns to the era cap - or set low enough to stop a run that was still
//--- improving. Neither is what "train to the best result" means. Quality is now enforced by
//--- WHICH era gets deployed (balanced-accuracy checkpoint ranking + this per-class recall
//--- floor) and WHEN a run ends (the plateau ladder), not by an accuracy threshold. Note the
//--- recall floor deliberately stays: it is not a performance target but the anti-collapse
//--- gate that makes auto-deploy safe.
m_objectiveMet = errorGateOK & & directionalRecallOK & & directionalRecallMeasured ;
}
// Only mark the persisted model "complete" once it actually converged this era -
// an interruption (stop) or an ordinary in-progress era must stay flagged incomplete
// so a restart resumes training instead of quietly treating a partial run as done.
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
// Convergence = "the plateau ladder is exhausted AND there is a recall-passing checkpoint
// to deploy" - the exact same condition the deploy branch beside the era-cap check uses, so
// the flag written into the .nnw here can never disagree with the decision to stop. While a
// run is still improving (or still has an escape stage left to try) this stays false and the
// per-era save correctly records an in-progress run. Previously this was
// (m_objectiveMet && m_oosStable), which needed the removed absolute accuracy target to mean
// anything: with that target gone m_oosStable alone - just 3 eras inside a 2pp band, which
// is true constantly - would have converged the run at the first flat spot.
m_trainingComplete = ( m_plateauStage > = PLATEAU_STAGE_DEPLOY ) & & m_bestPassedRecall & & m_haveOosCheckpoint ;
double currentIndicatorParams [ ] ;
m_indicatorTuner . Flatten ( currentIndicatorParams ) ;
if ( ! Net . Save ( m_activeFileName + " .nnw " , dError , dUndefine , dForecast , dtStudied , m_activeFileCommon , m_eraCount , m_trainingComplete , currentIndicatorParams ) )
Print ( __FUNCTION__ + " : ERROR - era-end Net.Save failed for " + m_activeFileName + " .nnw (era " + IntegerToString ( m_eraCount ) + " ). Training continues but this era's checkpoint was NOT persisted - a crash/restart now would resume from an older era. " ) ;
if ( ! SaveModelStats ( m_activeFileName , m_activeFileCommon ) ) // keep calibration state paired with the just-saved weights
Print ( __FUNCTION__ + " : ERROR - SaveModelStats failed for " + m_activeFileName + " (era " + IntegerToString ( m_eraCount ) + " ). Calibration/online-learning state not persisted this era. " ) ;
SaveShadowNet ( currentIndicatorParams ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
}
if ( shouldLogProgress )
{
string recallInfo = ( logBuyRecallPct < 0 & & logSellRecallPct < 0 & & logNeutralRecallPct < 0 ) ? " " :
( " | OOS recall Buy: " + ( logBuyRecallPct < 0 ? " n/a " : IntegerToString ( logBuyRecallPct ) + " % " ) +
" Sell: " + ( logSellRecallPct < 0 ? " n/a " : IntegerToString ( logSellRecallPct ) + " % " ) +
" Neutral: " + ( logNeutralRecallPct < 0 ? " n/a " : IntegerToString ( logNeutralRecallPct ) + " % " ) +
" (need >= " + IntegerToString ( m_minDirectionalRecallPct ) + " % each) " ) ;
//--- Balanced accuracy = the checkpoint-selection metric (see m_bestBalancedOos). Shown so the
//--- number the deployed model is actually chosen on is visible next to the recalls it averages.
feat(ai): rank checkpoints on directional precision, not balanced accuracy
Balanced accuracy is maximized by exactly the model this system must never
deploy. Measured frontier at fixed signal strength, base rate 6.1%:
tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0%
tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3%
tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5%
It rises monotonically as the model calls MORE and is right LESS, because
two of its three terms are directional recalls that a call-everything model
drives to ~95%, while the Neutral term it sacrifices counts for only a
third. The 2026-07-29 run landed exactly there: balanced 58-64% while
calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base
rate. Only the per-class recall floor stopped those deploying - a guard
doing the job the objective should have been doing - and that same guard
also rejected the genuinely useful sparse-but-precise checkpoints.
Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how
many were right. That is what a trading edge is. Two anti-degenerate floors
bracket it, since precision alone is trivially maximized by calling almost
nothing: coverage must reach a fraction of the true directional base rate
(derived, not configured - it adapts to any symbol/timeframe/label rule),
and precision must at least beat that base rate.
Against the same frontier the deploy order inverts from
tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first)
to
tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage)
Balanced accuracy is kept in the log as a diagnostic and marked as such, so
a run where the two disagree - the signature of an over-caller - is visible
at a glance. MinRecall no longer decides what ships; it now only drives the
diagnostic recall line and is a candidate for removal.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
//--- Balanced accuracy is retained as a DIAGNOSTIC only - selection ranks on directional
//--- precision now (see the SELECTION METRIC note). Both are shown so a run where they
//--- disagree - the signature of an over-calling model - is visible at a glance.
string balancedInfo = ( logBalancedAccPct < 0 ) ? " " : ( " | OOS balanced acc " + IntegerToString ( logBalancedAccPct ) + " % (diagnostic) " ) ;
string selectionInfo = ( logDirPrecPct < 0 ) ? " | SELECT: no directional calls " :
( " | SELECT dir-precision " + IntegerToString ( logDirPrecPct ) + " % on " +
fix: refuse invalid SL/TP, fix the unreachable deploy floor, scale the horizon
Three defects found by reading the 2026-08-01 training logs, all of which
only became visible because the relabel made the numbers mean something.
1. A STALE ENUM TRAINED FOUR MODELS ON THE WRONG TARGET.
`OnInit: trade settings snapshot - SL_Mode=1 TP_Mode=-101`
-101 was TP_PREV_SWING, deleted from TAKE_PROFIT_MODE on 2026-07-31 in
7eb48f5. MetaTrader does not validate a saved enum input against the
enum's current members, so charts saved before that kept the old
integer. BarrierMultiples()'s `if(tpMult <= 0.0) tpMult = slMult;`
then quietly turned it into a 1:1 barrier, and all four topologies
trained ~250 eras against a strategy nobody selected - while the log
reported "target 1.00*ATR" as though it were configured.
Since the relabel these two inputs ARE the label definition, so this
is not a bad trade setting, it is a wrong dataset. ValidateBarrier-
Inputs() now refuses to start (INIT_FAILED + Alert + an explicit fix)
on any value that is not an enum member. Members are enumerated rather
than range-checked because both enums are sparse and carry negative
sentinels, so no min/max test can tell a legal value from a deleted
one - which is the entire failure mode. The fallback survives as
belt-and-braces but now announces itself: a fallback that cannot say
it fired is indistinguishable from correct behaviour.
2. THE DEPLOYABILITY FLOOR BECAME MATHEMATICALLY UNREACHABLE.
`tradeableOK` required `dirPrecPct >= baseRatePct`, where baseRatePct
is Buy+Sell as a share of all bars. At the old exact-pivot target that
was ~6%, so "beat the base rate" read as "beat chance" and the test
looked sound. Triple-barrier labels put it at ~83%, so the gate now
demanded 83% directional precision - impossible by construction.
Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing
safe to deploy" at a perfectly healthy 43-45% precision, with no
checkpoint able to ship however good it got.
Replaced with ZERO-SKILL precision, max(Buy,Sell)/allBars: exactly the
score of the degenerate always-call-one-direction model this floor
exists to reject. Correct at any base rate - ~43% on the current
labels, ~3% on the old rare-pivot ones. The era line now prints
"(chance N%, edge +Mpp)" beside the selection score, because 44%
precision is excellent against a 3% chance level and worthless against
a 43% one, and reading the first as the second is what made tonight's
run look better than it was.
3. THE HORIZON IGNORED THE BARRIER GEOMETRY.
ComputeBarrierHorizonBars() returned the median ZigZag leg, which
measures how long a ~1 ATR move takes and says nothing about how long
the CONFIGURED barrier needs. First-passage time out of [-m,+k] scales
with m*k, so a 1:3 barrier takes ~3x as long as 1:1; the unscaled
horizon would have timed out most 1:3 trades and pushed Neutral
straight back up, re-creating the imbalance the relabel removes.
Now multiplied by slMult*tpMult, calibrated against a real measurement
rather than assumed: the accidental 1:1 run resolved at horizon 12 with
only 16.7% timeouts, so the swing median is the right scale at m*k=1.
Verifiable, not just asserted: the prebuild now counts barriers that
ended on the VERTICAL barrier and reports them as a share of Neutral.
Neutral conflates "timed out" with "stopped out" and only the first
indicts the horizon.
Both builds compile 0 errors / 0 warnings. Forces a retrain - correcting
TP_Mode re-keys the fingerprint (|TB:1:-101 -> |TB:1:3), which is right:
no existing model was trained on the intended target.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:30:49 -04:00
IntegerToString ( logCoveragePct ) + " % of bars " +
( logChancePrecPct > = 0
? " (chance " + IntegerToString ( logChancePrecPct ) + " %, edge " +
( logDirPrecPct - logChancePrecPct > = 0 ? " + " : " " ) +
IntegerToString ( logDirPrecPct - logChancePrecPct ) + " pp) "
: " " ) ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// See logBuyPredPct's declaration comment for why this is worth logging alongside recall -
// it's what tells apart a suppressed/dead output (predicted rate stuck at 0%) from a
// miscalibrated boundary (predicted rate healthy, precision poor), which look identical from
// recall alone.
string predictedInfo = ( logBuyPredPct < 0 & & logSellPredPct < 0 ) ? " " :
( " | OOS calls Buy: " + ( logBuyPredPct < 0 ? " n/a " : IntegerToString ( logBuyPredPct ) + " % " ) +
" (win rate " + ( logBuyPrecPct < 0 ? " n/a " : IntegerToString ( logBuyPrecPct ) + " % " ) + " ) " +
" Sell: " + ( logSellPredPct < 0 ? " n/a " : IntegerToString ( logSellPredPct ) + " % " ) +
" (win rate " + ( logSellPrecPct < 0 ? " n/a " : IntegerToString ( logSellPrecPct ) + " % " ) + " ) " ) ;
//--- Live-fired precision: the number that actually predicts forward-trading performance - only
//--- the directional calls that cleared the confidence floor under the live/prior-corrected rule
//--- (see AdjustedSignalFromSoftmax). Count in parentheses = how many bars the model would have
//--- traded this era. "0" fires = the calibration is (this era) suppressing all directional trades.
string liveInfo = ( m_lastBuyFired < = 0 & & m_lastSellFired < = 0 ) ? " | live fires 0 this era " :
( " | live win rate Buy: " + ( logBuyFiredPrecPct < 0 ? " n/a " : IntegerToString ( logBuyFiredPrecPct ) + " % " ) +
" ( " + IntegerToString ( m_lastBuyFired ) + " ) " +
" Sell: " + ( logSellFiredPrecPct < 0 ? " n/a " : IntegerToString ( logSellFiredPrecPct ) + " % " ) +
" ( " + IntegerToString ( m_lastSellFired ) + " ) " ) ;
feat(ai): measure precision per confidence tier; fix stale metric labels
Two things the 2026-07-30 run exposed.
1. Every user-facing message still called the selection metric "balanced
accuracy". It has ranked on directional precision since a142749, so
"CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5%
PRECISION as if it were macro-recall, while the same era logged an
actual balanced accuracy of 49%. Two different numbers under one
name, in the line that announces a deploy. Relabelled at every site,
including the stage-3 refusal, which still described the per-class
recall floor that stopped being the gate.
2. Precision is now bucketed by confidence tier and logged per era,
both per-tier and cumulatively from each tier upward:
| tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ...
The per-tier number says whether confidence is calibrated to
correctness at all; if it does not rise T0->T3, raising the floor
buys nothing and that is the finding. The ">=" number is what a floor
would actually deliver, with its fire count, so the coverage cost is
visible in the same line. Tier weights are 25/50/75/100, so for an
AI-only config Min_Vote_Open maps straight across: 50 = ">=T1",
75 = ">=T2", 100 = ">=T3".
Bucketing happens at the existing live-fired accounting site, so it
measures exactly the population that trades - not the raw argmax.
Both builds compile 0 errors, 0 warnings. No retrain needed for either.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
//--- Precision BY CONFIDENCE TIER, and cumulatively from each tier upward - the two numbers a
//--- decision about Min_Vote_Open actually needs. The per-tier figure says whether confidence is
//--- calibrated to correctness at all (it should rise T0->T3; if it does not, raising the floor
//--- buys nothing and the finding is that the head's confidence is uninformative). The ">=Tn"
//--- figure is what you would ACTUALLY get, because a floor keeps every tier at or above it, and
//--- it comes with the fire count so the coverage cost of raising the floor is visible in the
//--- same line. Tier weights are 25/50/75/100, so for an AI-only config the input maps straight
//--- across: Min_Vote_Open 50 = ">=T1", 75 = ">=T2", 100 = ">=T3".
string tierInfo = " " ;
int tierFiredTotal = 0 ;
for ( int ti = 0 ; ti < 4 ; ti + + )
tierFiredTotal + = m_oosTierFired [ ti ] ;
if ( tierFiredTotal > 0 )
{
tierInfo = " | tier prec " ;
for ( int ti = 0 ; ti < 4 ; ti + + )
{
int cumFired = 0 , cumHits = 0 ;
for ( int tj = ti ; tj < 4 ; tj + + )
{
cumFired + = m_oosTierFired [ tj ] ;
cumHits + = m_oosTierHits [ tj ] ;
}
tierInfo + = " T " + IntegerToString ( ti ) + " : " +
( m_oosTierFired [ ti ] > 0
? IntegerToString ( ( int ) MathRound ( 100.0 * m_oosTierHits [ ti ] / m_oosTierFired [ ti ] ) ) + " % "
: " n/a " ) +
" ( " + IntegerToString ( m_oosTierFired [ ti ] ) + " ) " +
( cumFired > 0
? " [>= " + IntegerToString ( ( int ) MathRound ( 100.0 * cumHits / cumFired ) ) + " %/ " +
IntegerToString ( cumFired ) + " ] "
: " " ) ;
}
}
2026-07-31 07:10:09 -04:00
// Per-layer weight movement. Pairs with rawOutInfo below: a collapsed constant-classifier state
// has two very different causes, and only this tells them apart. If every layer moves and the
// output still collapses, the architecture or the objective is at fault; if one stage sits at
// ~0.000% era after era while the others move, that stage is receiving no gradient and no amount
// of retraining or hyperparameter work will help. See CNet::LayerLearningReport.
string layerInfo = ( CheckPointer ( Net ) = = POINTER_INVALID ) ? " " :
( " | dW/W " + Net . LayerLearningReport ( ) ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// Raw-output saturation diagnostic - see m_oosOutMin's declaration comment. Spread ~0 with
// all six min/max values pinned together = the collapsed constant-classifier state.
string rawOutInfo = ( m_oosOutCount < = 0 ) ? " " :
StringFormat ( " | OOS raw out B:%.3f..%.3f S:%.3f..%.3f N:%.3f..%.3f spread avg %.4f " ,
m_oosOutMin [ 0 ] , m_oosOutMax [ 0 ] , m_oosOutMin [ 1 ] , m_oosOutMax [ 1 ] ,
m_oosOutMin [ 2 ] , m_oosOutMax [ 2 ] , m_oosOutSpreadSum / m_oosOutCount ) ;
//--- No "(target X%)" any more - there is no absolute accuracy target. What replaces it as the
//--- progress indicator is the plateau counter: how many eras since the last new best, and how
//--- close that is to ending the run (see the PLATEAU_* ladder).
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
string plateauInfo = ( m_bestBalancedOos < 0 ) ? " " :
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
( " | best bal " + DoubleToString ( m_bestBalancedOos , 1 ) + " %, " + IntegerToString ( m_erasSinceBestBalanced ) +
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
" eras since (stage " + IntegerToString ( m_plateauStage ) + " / " + IntegerToString ( PLATEAU_STAGE_DEPLOY ) + " ) " ) ;
fix(ui): unique chart tag, product-grade panel, responsive under load
Three separate reports from one deploy.
1. CONV, LSTM and HYBRID all came back tagged [4109]. The weights
fingerprint omits the topology type on purpose - the file path already
separates it (State\CONV\ vs State\LSTM\ vs State\HYB\) and hashing a
value that is constant within a folder buys nothing while re-keying
every trained model into a forced retrain. So the files were never at
risk, but the tag could not do its one job. Prefixing the short id
makes it unique on the display side only; the hex half still greps
straight to the .nnw inside the folder the prefix names.
2. The default panel read like a training console. Six lines down to
three, each answering a question an owner actually has. The deploy
internals (best score, eras-since-best, ladder stage) were developer
diagnostics describing a recall floor that no longer decides anything,
and were already in the era-end journal line. In-sample accuracy left
the panel too: it grades the model on bars it trained on, so it always
flatters, and showing it beside the honest number invites reading the
wrong one. New compile-time DebuggingMode constant - deliberately not
an input - carries the IS/OOS pair and the resolved model path into
the journal instead. No extra Inputs row, no extra Market description
line, no user-reachable firehose.
3. Panel drag and buttons stuttered under training load, exactly as the
2026-07-26 note raising the chunk budget to 200ms warned they might.
Backed off to the documented 120ms - worst-case click latency is that
budget - and the derived topology (~292k weights to ~29k) makes the
throughput this costs far cheaper than when that note was written.
Also halved the panel redraw rate to 2.5 Hz: ChartRedraw repaints the
whole chart, so its cost scales with accumulated arrows, and 5 Hz was
the larger half of the stutter. Era-end still force-refreshes.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 09:05:58 -04:00
//--- Lifetime IS/OOS directional accuracy. The panel now shows the out-of-sample half alone (see
//--- ComputeCompoundedAccuracyLine - the in-sample figure grades the model on bars it trained on,
//--- so it always reads higher than anything forward trading will deliver and does not belong on
//--- a product's face). The GAP between the two is still the over-fitting read, so it survives
//--- here, once per era, behind the compile-time DebuggingMode constant.
string lifetimeInfo = ( ! DebuggingMode | | ( m_cumIsTotal < = 0 & & m_cumOosTotal < = 0 ) ) ? " " :
( " | lifetime dir acc IS " + ( m_cumIsTotal > 0 ? IntegerToString ( ( int ) MathRound ( m_cumIsCorrect * 100.0 / m_cumIsTotal ) ) + " % " : " n/a " ) +
" OOS " + ( m_cumOosTotal > 0 ? IntegerToString ( ( int ) MathRound ( m_cumOosCorrect * 100.0 / m_cumOosTotal ) ) + " % " : " n/a " ) +
" over " + IntegerToString ( m_cumIsTotal + m_cumOosTotal ) + " calls " ) ;
2026-07-31 07:10:09 -04:00
Print ( ID + " : training in progress - era " + IntegerToString ( m_eraCount ) + " , OOS accuracy " + DoubleToString ( dOosForecast , 1 ) + " %, IS error " + DoubleToString ( dError , 2 ) + recallInfo + balancedInfo + selectionInfo + predictedInfo + liveInfo + tierInfo + plateauInfo + lifetimeInfo + rawOutInfo + layerInfo ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// Forced (unthrottled) panel refresh, right here alongside the console line above, using this
// era's own just-finalized m_eraCount/dOosForecast - see UpdateTrainingStatusLabel's
// declaration comment for why this can't just rely on the next throttled bar-scan call to
// catch up (it would, but a full era later than the console already reported it).
UpdateTrainingStatusLabel ( " Era complete " , m_lastDisplayNeuron0 , m_lastDisplayNeuron1 , m_lastDisplayNeuron2 , m_lastDisplaySignal , true ) ;
}
//--- Genuine convergence THIS era (not a stale m_trainingComplete carried over from a previous
//--- run) - (re)start the evaluation-only continual-learning OOS walk. Always rebuilt fresh from
//--- the just-converged weights; never resumes a stale walk from a superseded model.
//--- m_trainingComplete is the plateau ladder's verdict now (see where it is assigned): "stopped
//--- improving after both escape attempts, and there is a recall-passing checkpoint to deploy".
//--- It replaces the old (m_objectiveMet && m_oosStable) test, which depended on the removed
//--- absolute accuracy target to mean anything - without it, m_oosStable alone (3 eras inside a 2pp
//--- band) would have declared convergence at the first flat spot in every run.
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
if ( ! stop & & m_trainingComplete )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
feat(ai): measure precision per confidence tier; fix stale metric labels
Two things the 2026-07-30 run exposed.
1. Every user-facing message still called the selection metric "balanced
accuracy". It has ranked on directional precision since a142749, so
"CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5%
PRECISION as if it were macro-recall, while the same era logged an
actual balanced accuracy of 49%. Two different numbers under one
name, in the line that announces a deploy. Relabelled at every site,
including the stage-3 refusal, which still described the per-class
recall floor that stopped being the gate.
2. Precision is now bucketed by confidence tier and logged per era,
both per-tier and cumulatively from each tier upward:
| tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ...
The per-tier number says whether confidence is calibrated to
correctness at all; if it does not rise T0->T3, raising the floor
buys nothing and that is the finding. The ">=" number is what a floor
would actually deliver, with its fire count, so the coverage cost is
visible in the same line. Tier weights are 25/50/75/100, so for an
AI-only config Min_Vote_Open maps straight across: 50 = ">=T1",
75 = ">=T2", 100 = ">=T3".
Bucketing happens at the existing live-fired accounting site, so it
measures exactly the population that trades - not the raw argmax.
Both builds compile 0 errors, 0 warnings. No retrain needed for either.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
Print ( ID + " : training CONVERGED at era " + IntegerToString ( m_eraCount ) + " - this is the best this configuration reached: dir-precision " +
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
DoubleToString ( m_bestBalancedOos , 1 ) + " %, blended OOS " + DoubleToString ( dOosForecast , 1 ) + " %, IS error " + DoubleToString ( dError , 2 ) +
2026-07-31 14:33:29 -04:00
" . No new best for " + IntegerToString ( m_erasSinceBestBalanced ) + " eras across " +
IntegerToString ( PLATEAU_STAGE_DEPLOY - 1 ) + " learning-rate warm restarts. " +
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
" Weights saved, switching to live inference. " ) ;
StartOosContinualSimulation ( bars , oosCutoff ) ;
}
if ( stop | | m_trainingComplete )
FinalizeTrainRun ( ) ;
//--- else: this era is done but the run continues - the next Train() call (re-triggered via
//--- ScheduleTrainingIfNeeded()'s custom event, same mechanism as always) starts the next era
//--- fresh, since m_eraResumePending is false while m_trainRunActive stays true
//--- Save this model's own learning-rate trajectory back out of the shared global before
//--- returning - see m_modelEta's declaration comment. Covers every path that reaches here
//--- (natural era completion, whether or not the run itself just finalized).
m_modelEta = eta ;
}
//+------------------------------------------------------------------+
//| Ends the current Train() run: restores the best-scoring era's |
//| checkpointed weights (if any beat the era the loop happened to |
//| end on), persists final state, and clears the resumable-run |
//| flags. Called both from Train() itself (natural stop/converge) |
//| and from StopTraining() (a mid-chunk Stop click won't get |
//| another "New Bar" event to resume into, since |
//| ScheduleTrainingIfNeeded() refuses to schedule while |
//| m_trainingStopRequested is set, so it must finalize synchronously |
//| there instead of being left dangling). |
//+------------------------------------------------------------------+
//| Era-cap decision: keep training (true) or deploy best + stop |
//| (false). Live chart -> operator dialog; headless -> stop. |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase : : PromptContinuePastEraCap ( double bestOos )
{
//--- No GUI in the Strategy Tester/optimizer - MessageBox() is unavailable there and would just
//--- stall a headless run, so deploy the best checkpoint found so far and stop (the safe default).
if ( MQLInfoInteger ( MQL_TESTER ) | | MQLInfoInteger ( MQL_OPTIMIZATION ) | | MQLInfoInteger ( MQL_FORWARD ) )
return false ;
//--- Reaching this cap is now the UNUSUAL outcome: a run normally ends itself when the plateau ladder
//--- runs out of escapes (see the PLATEAU_* constants), which is a statement about the run having
//--- stopped improving rather than about any accuracy number. So the interesting question here is why
//--- the ladder had not finished yet - either the run was still finding new bests (just needs more
//--- eras), or nothing has ever cleared the per-class recall floor, which blocks auto-deploy on
//--- purpose so a one-class model can never ship. Spell out which.
bool recallMet = ( m_lastBuyRecallPct < 0 | | m_lastBuyRecallPct > = m_minDirectionalRecallPct ) & &
( m_lastSellRecallPct < 0 | | m_lastSellRecallPct > = m_minDirectionalRecallPct ) ;
string neutralNote = ( m_priorNeutral > 0.0 )
? ( " inflated by the ~ " + IntegerToString ( ( int ) MathRound ( m_priorNeutral * 100.0 ) ) + " % Neutral base rate " )
: " inflated by the dominant Neutral class " ;
string reasons = " " ;
if ( ! m_bestPassedRecall )
reasons + = " - No era has ever cleared the per-class recall floor, so there is no model safe to \n " +
" auto-deploy yet (a model that ignores Buy or Sell must never ship) \n " ;
else
reasons + = " - Still improving: " + IntegerToString ( m_erasSinceBestBalanced ) + " eras since the last new best, plateau stage " +
IntegerToString ( m_plateauStage ) + " of " + IntegerToString ( PLATEAU_STAGE_DEPLOY ) + " (the run ends itself at stage " +
IntegerToString ( PLATEAU_STAGE_DEPLOY ) + " ) \n " ;
if ( ! recallMet )
reasons + = " - Latest era's per-class recall below the floor: Buy " +
( m_lastBuyRecallPct < 0 ? " n/a " : IntegerToString ( m_lastBuyRecallPct ) + " % " ) + " / Sell " +
( m_lastSellRecallPct < 0 ? " n/a " : IntegerToString ( m_lastSellRecallPct ) + " % " ) +
" (need >= " + IntegerToString ( m_minDirectionalRecallPct ) + " % each) \n " ;
if ( ! m_objectiveMet )
reasons + = " - The latest era did not produce a valid model (recall floor not met/not measured) \n " ;
string balancedStr = ( m_bestBalancedOos > 0.0 )
feat(ai): measure precision per confidence tier; fix stale metric labels
Two things the 2026-07-30 run exposed.
1. Every user-facing message still called the selection metric "balanced
accuracy". It has ranked on directional precision since a142749, so
"CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5%
PRECISION as if it were macro-recall, while the same era logged an
actual balanced accuracy of 49%. Two different numbers under one
name, in the line that announces a deploy. Relabelled at every site,
including the stage-3 refusal, which still described the per-class
recall floor that stopped being the gate.
2. Precision is now bucketed by confidence tier and logged per era,
both per-tier and cumulatively from each tier upward:
| tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ...
The per-tier number says whether confidence is calibrated to
correctness at all; if it does not rise T0->T3, raising the floor
buys nothing and that is the finding. The ">=" number is what a floor
would actually deliver, with its fire count, so the coverage cost is
visible in the same line. Tier weights are 25/50/75/100, so for an
AI-only config Min_Vote_Open maps straight across: 50 = ">=T1",
75 = ">=T2", 100 = ">=T3".
Bucketing happens at the existing live-fired accounting site, so it
measures exactly the population that trades - not the raw argmax.
Both builds compile 0 errors, 0 warnings. No retrain needed for either.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
? ( " \n Best directional precision, coverage-weighted (the metric the deployed \n checkpoint is chosen on): " + DoubleToString ( m_bestBalancedOos , 1 ) +
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
" % \n Best blended OOS accuracy: " + DoubleToString ( bestOos , 1 ) + " % ( " + neutralNote + " ) \n " )
: " " ;
string msg = ID + " : training reached the " + IntegerToString ( m_maxErasPerRun ) +
" -era cap before it finished on its own. \n \n " +
" Training now runs until it stops improving, then deploys its best model. Status: \n " +
reasons +
balancedStr +
" \n Continue training? \n \n " +
" Yes = keep training for another " + IntegerToString ( m_maxErasPerRun ) + " eras \n " +
" No = deploy the best checkpoint so far and stop training " ;
int res = MessageBox ( msg , " Warrior EA - training " , MB_YESNO | MB_ICONQUESTION ) ;
return ( res = = IDYES ) ;
}
//+------------------------------------------------------------------+
//+------------------------------------------------------------------+
//| See the declaration comment - the single deploy-persistence path. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase : : PersistDeployedModel ( void )
{
if ( CheckPointer ( Net ) = = POINTER_INVALID )
return ;
double currentIndicatorParams [ ] ;
m_indicatorTuner . Flatten ( currentIndicatorParams ) ;
if ( ! Net . Save ( m_activeFileName + " .nnw " , dError , dUndefine , dForecast , dtStudied , m_activeFileCommon , m_eraCount , m_trainingComplete , currentIndicatorParams ) )
Print ( __FUNCTION__ + " : ERROR - Net.Save failed for " + m_activeFileName + " .nnw. The deployed model was NOT persisted to disk. " ) ;
//--- Deploy-time gate: does this model's pure-MQL5 forward pass match the backend? If so, an
//--- inference-only backtest can run DLL-free (see ValidateCpuInference / CNet::SetCpuInference).
//--- Persisted into the .stats written next. Chart-only; safe-false everywhere else.
m_mqlInferenceValidated = ValidateCpuInference ( ) ;
if ( ! SaveModelStats ( m_activeFileName , m_activeFileCommon ) ) // keep calibration state paired with the just-saved weights
Print ( __FUNCTION__ + " : ERROR - SaveModelStats failed for " + m_activeFileName + " . Calibration state not persisted. " ) ;
SaveShadowNet ( currentIndicatorParams ) ;
}
//+------------------------------------------------------------------+
void CExpertSignalAIBase : : FinalizeTrainRun ( void )
{
//--- deploy the most stable/best-scoring era's weights rather than whatever the run happened to
//--- end on (which may reflect drift after the objective was first hit, or an aborted run). Restore
//--- is now the in-MEMORY snapshot (CNet::RestoreWeights) - see CaptureWeights' note for why the old
//--- file-based restore couldn't work on the CPU-DLL backend.
if ( m_haveOosCheckpoint )
{
if ( Net . RestoreWeights ( ) )
{
dOosForecast = m_bestOosForecast ;
RefreshLatestSignal ( ) ;
fix(deinit): a full model write was running ahead of the cheap cleanup
"Abnormal termination" is back, and this time it is not the arrows. The
timing names the culprit exactly:
16:02:31.547 OnDeinit: shutting down
16:02:36.003 Abnormal termination <- 4.46 s, MetaTrader gave up
16:02:36.226 chart signals - persisted <- cleanup finished 0.2 s LATE
OnDeinit called StopTraining() BEFORE the chart cleanup. StopTraining()
finalises an in-flight run, and FinalizeTrainRun() restores the best
checkpoint and then persists it - a full ~1MB model write per signal. So
the expensive step ran ahead of the cheap bounded one, which is precisely
the inversion the shutdown ordering exists to prevent. The previous fix
put PersistWeightsOnShutdown last and missed that StopTraining smuggles a
second save in at the front.
Two changes:
Cleanup now runs FIRST, then StopTraining, then the weight save. The
visible teardown is cheap and bounded, so it always completes even when
everything after it is killed.
And the deploy-persist inside FinalizeTrainRun is suppressed during
shutdown. RestoreWeights() is an in-MEMORY swap, so the best checkpoint
is already the live net by that line, and PersistWeightsOnShutdown writes
exactly those weights moments later. The old path wrote the same model
twice per signal - eight full writes across four charts - for no benefit.
A user-pressed Stop still persists immediately, because nothing else
would.
Compiles 0 errors / 0 warnings. Build tag deinit-order-v2.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 16:06:40 -04:00
//--- NOT during shutdown. RestoreWeights() above is an in-MEMORY swap, so the best checkpoint is
//--- already the live net by this line - and OnDeinit's PersistWeightsOnShutdown() is about to
//--- write exactly those weights anyway. Persisting here too means TWO full ~1MB model writes per
//--- signal on the shutdown path, ahead of the chart cleanup, which is what put OnDeinit over
//--- MetaTrader's budget: measured 4.46 s to "Abnormal termination" on 2026-08-01, with the chart
//--- cleanup completing 0.2 s AFTER the kill. Nothing is lost by skipping it; the same bytes reach
//--- the same file one call later.
if ( ! m_shutdownInProgress )
PersistDeployedModel ( ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
}
//--- Clean up any legacy on-disk checkpoint from an older (file-based) build so it can't linger.
int checkpointFlags = m_activeFileCommon ? FILE_COMMON : 0 ;
if ( FileIsExist ( m_activeFileName + " _ckpt.tmp " , checkpointFlags ) )
FileDelete ( m_activeFileName + " _ckpt.tmp " , checkpointFlags ) ;
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- (dtStudied used to be held back while scoring a throwaway candidate - that marker belongs
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- to the DEPLOYED model's "studied up to" state; a candidate eval must leave it untouched. The
//--- checkpoint block above is already inert in eval mode (m_haveOosCheckpoint stays false).
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
if ( m_eraCount > 0 )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
dtStudied = m_lastBarTime ;
m_trainRunActive = false ;
m_eraResumePending = false ;
m_haveOosCheckpoint = false ;
//--- Persist the arrows now drawn on the chart so a deploy/stop survives a later re-add/recompile
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- without a retrain (durable even if the terminal never gets a clean OnDeinit).
fix(deinit): O(n^2) arrow prune blew the shutdown budget and littered 3 charts
Reported as "the perceptron correctly cleaned its chart on deinit, the
other 3 did not, abnormal termination". Measured from the 2026-08-01 log,
time from "OnDeinit: shutting down" to MetaTrader force-terminating:
PAI 3.75 s -> survived, chart cleaned
CONV 4.71 s -> Abnormal termination
LSTM 4.28 s -> Abnormal termination
HYBRID 4.16 s -> Abnormal termination
In all four the last line printed is the inference census, which is the
end of StopTraining() - so the overrun is inside ShutdownChartCleanup(),
i.e. between saving the arrows and purging them.
The cost is the prune loop at the end of SaveChartSignals():
for(int i = 0; i < prunedCount; i++)
ObjectDelete(0, SIG_ARROW_PREFIX + TimeToString(pruned[i]));
ObjectDelete is O(objects) on a crowded chart, so this is O(n^2). It was
harmless while the model called a direction on ~6% of bars. After the
triple-barrier relabel the models call on 83-94% of bars, the chart
carries many thousands of arrows, and the loop overran MetaTrader's
OnDeinit budget - so PurgeChart() never ran and the arrows stayed on
screen. The slow tidy-up starved the fast one.
The work was pure waste at that moment: ShutdownChartCleanup purges every
arrow with a single bulk ObjectsDeleteAll immediately afterwards.
Deleting them one at a time first has no effect except to prevent the
bulk delete from happening at all.
SaveChartSignals takes a pruneChartObjects flag, and the two shutdown
call sites pass false:
- ShutdownChartCleanup passes `preserveChartArrows`, which is exactly
right: prune when the arrows are STAYING (chart and sidecar must
agree), skip when they are about to be purged wholesale.
- FinalizeTrainRun passes !m_trainingStopRequested. Removing a chart
MID-ERA reaches StopTraining -> FinalizeTrainRun, which took the
expensive path a second time, even earlier, before anything had been
cleared. Same defect one call site up; it only escaped notice because
the observed removals happened to land between eras.
Normal convergence and the live per-era path are unchanged - they still
prune, which is what keeps the chart object count bounded.
This also restores the invariant the 2026-07 fix intended ("chart cleanup
runs BEFORE the heavy weight save so a stall cannot leave the chart
littered"). That fix moved cleanup ahead of the WEIGHT save, but cleanup
had since grown its own slow step ahead of its own fast one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:38:36 -04:00
//--- The prune is suppressed when a STOP is in flight, because that means StopTraining() called us and
//--- ShutdownChartCleanup() is about to bulk-purge every arrow anyway. Without this, removing a chart
//--- MID-ERA takes the expensive path twice: once here and once in the cleanup that follows, both
//--- before anything has been cleared. Same defect as the shutdown prune, one call site earlier - see
//--- the prune block in SaveChartSignals() for the measurement.
SaveChartSignals ( ! m_trainingStopRequested ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
# endif // WARRIOR_AIBASE_TRAINING_MQH