refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//+------------------------------------------------------------------+
//| Warrior_EA |
//| AnimateDread |
//| |
//| Live continual learning, the EMA shadow net, and the OOS continu|
//| |
//| PARTIAL IMPLEMENTATION FILE - not standalone. |
//| This holds CExpertSignalAIBase method BODIES only. The class |
//| declaration lives in Expert\ExpertSignalAIBase.mqh, which |
//| #includes this file at the bottom, after the declaration. Do not |
//| include it anywhere else and do not compile it on its own. |
//| |
//| Split out purely to make the 8216-line original navigable; the |
//| code inside was moved verbatim, not rewritten. |
//+------------------------------------------------------------------+
# ifndef WARRIOR_AIBASE_ONLINELEARNING_MQH
# define WARRIOR_AIBASE_ONLINELEARNING_MQH
//+------------------------------------------------------------------+
//| Clones the just-converged Net into a separate CNet (m_simOosNet) |
//| and arms a chunked bar-by-bar walk through the OOS window - see |
//| AdvanceOosSimulationChunk(). Evaluation-only: the clone's learned |
//| weights are never written back to Net or any persisted file. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase : : StartOosContinualSimulation ( int bars , int oosCutoff )
{
if ( m_simOosRunActive )
{
delete m_simOosNet ;
m_simOosNet = NULL ;
m_simOosRunActive = false ;
}
if ( oosCutoff < = 0 )
return ; // nothing to walk this run
//--- Clone via the full Save()/Load() pair. Load() calls InitOpenCL()/InitDirectML() before
//--- reconstructing layers, so a bare "new CNet(NULL)" ends up with a GPU/DirectML backend matching
//--- production. Any lighter-weight restore that reused the CALLER's opencl/directml pointers would
//--- be wrong here: on a fresh CNet(NULL) (whose constructor no-ops for a NULL description) those are
//--- unset, and the clone would come out degenerate. (A file-based checkpoint pair used to sit beside
//--- Save/Load and had exactly that flaw; it has been removed - the in-run snapshot is now the
//--- in-memory CNet::CaptureWeights/RestoreWeights.)
//--- Co-locate this ephemeral clone temp with the active model (COMMON on a live chart, LOCAL in the
//--- tester sandbox) instead of always LOCAL. Uses m_activeFileName for the same reason (the active
//--- model's base name, whichever context we're in).
string simFile = m_activeFileName + " _simoos.tmp " ;
int simFlags = m_activeFileCommon ? FILE_COMMON : 0 ;
double ip [ ] ;
if ( ! Net . Save ( simFile , 0.0 , 0.0 , 0.0 , dtStudied , m_activeFileCommon , m_eraCount , m_trainingComplete , ip ) )
return ;
m_simOosNet = new CNet ( NULL ) ;
double loadE , loadU , loadF ;
datetime loadTime ;
long loadEra ;
bool loadComplete ;
double loadIp [ ] ;
bool loaded = m_simOosNet . Load ( simFile , loadE , loadU , loadF , loadTime , m_activeFileCommon , loadEra , loadComplete , loadIp , true /*quiet: this evaluation-only sim is optional - on a miss it simply doesn't run*/ ) ;
FileDelete ( simFile , simFlags ) ;
if ( ! loaded )
{
delete m_simOosNet ;
m_simOosNet = NULL ;
return ;
}
m_simOosCutoff = oosCutoff ;
m_simOosBarIndex = oosCutoff - 1 ;
m_simOosForecast = 0 ;
m_simOosSamples = 0 ;
m_simOosRunActive = true ;
}
//+------------------------------------------------------------------+
//| Advances the evaluation-only continual-learning OOS walk by up |
//| to TRAIN_TIME_BUDGET_MS of work, then yields (same chunking |
//| pattern as the real era loop's m_eraResumePending). For each bar, |
//| oldest-OOS to newest: predict with the clone's CURRENT weights, |
//| score against the cached true label, THEN let the clone learn |
//| from it (single pass, no oversampling replay) - simulating how |
//| the model would adapt bar-by-bar in real forward trading. Never |
//| touches Net, never Saves the clone - purely an evaluation metric.|
//+------------------------------------------------------------------+
void CExpertSignalAIBase : : AdvanceOosSimulationChunk ( void )
{
const uint SIM_TIME_BUDGET_MS = 80 ;
uint chunkStartTick = GetTickCount ( ) ;
//--- Mirror OnlineLearnStep()'s pinned rate for the duration of this chunk, and hand the shared
//--- global back on BOTH exit paths - this simulation is only a valid forecast of live continual
//--- learning if it steps at the same size, and `eta` is shared by every signal instance.
double savedEta = eta ;
eta = m_modelEta * ONLINE_LEARN_ETA_SCALE ;
int i ;
for ( i = m_simOosBarIndex ; i > = 0 ; i - - )
{
if ( GetTickCount ( ) - chunkStartTick > = SIM_TIME_BUDGET_MS )
{
m_simOosBarIndex = i ;
eta = savedEta ;
return ;
}
if ( i > = ArraySize ( m_labelCacheHasValue ) | | ! m_labelCacheHasValue [ i ] )
continue ; // no cached label for this bar (e.g. right at a window edge) - nothing to learn from
//--- Window ends AT (includes) bar i - see Train()'s matching r declaration comment for why.
int r = i ;
fix: the sequence models were reading the window backwards
BuildFeatureWindow() replaces eight hand-rolled copies of the same loop
and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first,
because MQL5 timeseries indices run backwards and `r + b` with b ascending
walks into the past.
Harmless for PAI and CONV - a dense layer learns a weight per position
either way, a conv learns time-mirrored kernels. Not harmless for the
recurrent stacks:
- LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t.
- It writes output[] only when t == steps-1: the visible output IS the
last hidden state.
- c_t = f*c_{t-1} + i*g decays toward the start of the sequence.
lstm_seq_flowcheck.cpp measured block 0's influence on the output at
1.2e-2 of block T-1's, at the shipped forget bias of 1.0.
So the bar being PREDICTED sat at the far end of the decay and the output
was handed to the OLDEST bar in the window - the exact inverse of what the
window is for. ~80x backwards on LSTM and HYBRID, on all three tiers
(OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never
surfaced as a backend discrepancy.
This does not create edge - the MI diagnostics read at the noise floor
(p=0.4975) with a working positive control. It makes the one hypothesis
those diagnostics explicitly do NOT cover testable: they are marginal and
per-bar, and state they "cannot rule out one that only exists in
combination or across time". The sequence model is the instrument for
across-time structure and it has been crippled, so that hypothesis has
never been honestly tested.
Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and
its features, so a stale .nnw would load cleanly and run a model fitted to
one ordering against the other, silently. Re-keying every config is the
point, not collateral damage. FORCES A FULL RETRAIN.
Also: the now-relative bar caches are re-keyed on the two live paths.
EnsureBarCachesCapacity() was only ever called from training paths, but
once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar
to RefreshConvergedSignal() and Train() is never re-entered - so nothing
cleared the feature cache again for the life of the process. A chart that
trained to convergence kept replaying the rows computed for the last
training era's bar grid: the live signal froze at its convergence-time
value, and OnlineLearnStep() backpropped those stale features against
freshly resolved labels. Backtests were never affected (an inference-only
process never allocates the arrays, so every read recomputes).
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 18:28:44 -04:00
if ( ! BuildFeatureWindow ( r ) )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
continue ;
m_simOosNet . feedForward ( TempData ) ;
m_simOosNet . getResults ( TempData ) ;
double simSignal = ( m_outputNeuronsCount = = 3 ) ? ApplyClassificationSoftmax ( ) : TempData [ 0 ] ;
//--- Pre-update softmax probabilities, read before TempData is rebuilt as the target vector -
//--- feeds the same alpha-balanced focal weight the live path applies (OnlineSampleWeight).
double sBuy = ( TempData . Total ( ) > 0 ) ? TempData . At ( 0 ) : 0.0 ;
double sSell = ( TempData . Total ( ) > 1 ) ? TempData . At ( 1 ) : 0.0 ;
double sNeutral = ( TempData . Total ( ) > 2 ) ? TempData . At ( 2 ) : 0.0 ;
bool buy = m_labelCacheBuy [ i ] ;
bool sell = m_labelCacheSell [ i ] ;
ENUM_SIGNAL trueSignal = buy ? Buy : ( sell ? Sell : Neutral ) ;
bool hit = ( DoubleToSignal ( simSignal ) = = trueSignal ) ;
m_simOosSamples + + ;
if ( hit )
m_simOosForecast + = ( 100 - m_simOosForecast ) / Net . recentAverageSmoothingFactor ;
else
m_simOosForecast - = m_simOosForecast / Net . recentAverageSmoothingFactor ;
TempData . Clear ( ) ;
if ( m_outputNeuronsCount = = 1 )
TempData . Add ( buy & & ! sell ? 1 : ! buy & & sell ? -1 : 0 ) ;
else
if ( m_outputNeuronsCount = = 3 )
{
TempData . Add ( buy ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW ) ;
TempData . Add ( sell ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW ) ;
TempData . Add ( ( ! buy & & ! sell ) ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW ) ;
}
m_simOosNet . backProp ( TempData , OnlineSampleWeight ( trueSignal , sBuy , sSell , sNeutral ) ) ;
}
eta = savedEta ;
delete m_simOosNet ;
m_simOosNet = NULL ;
m_simOosRunActive = false ;
Print ( ID + " : continual-learning OOS simulation complete - " + IntegerToString ( m_simOosSamples ) + " samples, accuracy " + DoubleToString ( m_simOosForecast , 1 ) + " % " ) ;
}
//+------------------------------------------------------------------+
2026-08-16 21:08:41 -04:00
//| Arms the one-shot pattern-database backfill (see the declaration |
//| comment). Called right after FinalizeTrainRun() has restored the |
//| DEPLOYED checkpoint, so the walk below scores with the exact |
//| weights that are about to trade live - not the last era's, which |
//| the plateau ladder may have superseded. |
//+------------------------------------------------------------------+
fix: drop the ranking slice for the calibration band; un-collapse the tiers
NOT COMPILED - user compiles.
(1) THE RANKING SLICE IS GONE. It reserved 20% of the OOS window so the
pattern-DB backfill would read bars the deployed checkpoint was not SELECTED on.
That objection stands; carving a new region to answer it did not. The calibration
band already has every property the slice was buying:
never trained on | never graded by pass 3 (which walks [0, oosCutoff) and so
never reaches it) | never seen by the deploy gate | purged by a full label
horizon on BOTH sides | and larger besides - 1,684 bars vs the ~970 carved
So the backfill now walks [calibLo, calibHi) and pass 3 goes back to grading the
entire OOS window, exactly as before any of this. The gate gets its full sample
back (~10% of a sigma), the split loses a region, and the failure mode found an
hour ago - a reserved region silently blanking ~10 months of chart arrows,
because arrows are only drawn on bars pass 3 grades - becomes impossible.
One impurity, stated in the completion log rather than hidden:
m_dirConfThreshold is FITTED on that band and the walk applies it to decide which
bars fired, so coverage there is mildly optimistic. One scalar under a coverage
floor, against checkpoint selection over hundreds of eras.
This backfill IS the deploy-time warm-up: it runs right after FinalizeTrainRun()
restores the deployed weights, so it scores with exactly what is about to trade.
(2) EVERY CALL WAS TIER 0, AND IT WAS ARITHMETIC. ConfidenceTier() quartiles
[floorConf, 1] where floorConf = 1/3 - the lowest magnitude a 3-way softmax
winner can hold. But it was fed CalibratedConfidenceMagnitude(), which multiplies
by m_confidenceCalScale, clamped to [0.3, 1.5]. That lower clamp is BELOW 1/3.
Whenever calibration bottoms out, t goes negative and MathMax(0, ...) pins every
call to tier 0.
Which is what the live run does. m_confidenceCalScale is EMA'd toward
empiricalAccuracy / avgClaimedConfidence; with the model over-calling Neutral,
3-class agreement sits near 10% against a claimed confidence near 0.9, so the
ratio is ~0.11 and clamps to 0.3 every era. Logged:
tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0)
828 calls, one bucket - the four tier weights and the entire per-tier pattern-DB
ranking reduced to a single number. The backfill was feeding a mechanism that
structurally could not rank.
Tiering now reads the RAW head magnitude, which genuinely lives on the
[1/3, 1] range these bounds were written for. Calibration keeps its real jobs -
AIConfidence() for MM sizing and SignedAIConfidence() for the vote are unchanged.
STILL OPEN, deliberately not touched here: the calibration TARGET itself.
empiricalAccuracy is 3-class agreement, which is the wrong quantity to scale a
DIRECTIONAL confidence against - it counts a Neutral class that is 0.19% of
labels. The honest target is the win rate on the calls the confidence describes
(directional precision), with the claimed-confidence average taken over those
same called bars. That needs a new accumulator and it interacts with the Neutral
over-calling being fixed elsewhere, so it wants one clean run first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 23:06:11 -04:00
void CExpertSignalAIBase : : StartPatternDatabaseBackfill ( int bars , int totalIter , int oosCutoff )
2026-08-16 21:08:41 -04:00
{
if ( m_dbBackfillDone | | m_dbBackfillActive )
return ;
if ( ! UseDatabaseRanking | | MQLInfoInteger ( MQL_TESTER ) | | MQLInfoInteger ( MQL_OPTIMIZATION ) | | MQLInfoInteger ( MQL_FORWARD ) )
return ;
if ( GetFilterID ( ) = = " NULL " | | oosCutoff < = 0 | | CheckPointer ( Net ) = = POINTER_INVALID )
return ;
fix: drop the ranking slice for the calibration band; un-collapse the tiers
NOT COMPILED - user compiles.
(1) THE RANKING SLICE IS GONE. It reserved 20% of the OOS window so the
pattern-DB backfill would read bars the deployed checkpoint was not SELECTED on.
That objection stands; carving a new region to answer it did not. The calibration
band already has every property the slice was buying:
never trained on | never graded by pass 3 (which walks [0, oosCutoff) and so
never reaches it) | never seen by the deploy gate | purged by a full label
horizon on BOTH sides | and larger besides - 1,684 bars vs the ~970 carved
So the backfill now walks [calibLo, calibHi) and pass 3 goes back to grading the
entire OOS window, exactly as before any of this. The gate gets its full sample
back (~10% of a sigma), the split loses a region, and the failure mode found an
hour ago - a reserved region silently blanking ~10 months of chart arrows,
because arrows are only drawn on bars pass 3 grades - becomes impossible.
One impurity, stated in the completion log rather than hidden:
m_dirConfThreshold is FITTED on that band and the walk applies it to decide which
bars fired, so coverage there is mildly optimistic. One scalar under a coverage
floor, against checkpoint selection over hundreds of eras.
This backfill IS the deploy-time warm-up: it runs right after FinalizeTrainRun()
restores the deployed weights, so it scores with exactly what is about to trade.
(2) EVERY CALL WAS TIER 0, AND IT WAS ARITHMETIC. ConfidenceTier() quartiles
[floorConf, 1] where floorConf = 1/3 - the lowest magnitude a 3-way softmax
winner can hold. But it was fed CalibratedConfidenceMagnitude(), which multiplies
by m_confidenceCalScale, clamped to [0.3, 1.5]. That lower clamp is BELOW 1/3.
Whenever calibration bottoms out, t goes negative and MathMax(0, ...) pins every
call to tier 0.
Which is what the live run does. m_confidenceCalScale is EMA'd toward
empiricalAccuracy / avgClaimedConfidence; with the model over-calling Neutral,
3-class agreement sits near 10% against a claimed confidence near 0.9, so the
ratio is ~0.11 and clamps to 0.3 every era. Logged:
tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0)
828 calls, one bucket - the four tier weights and the entire per-tier pattern-DB
ranking reduced to a single number. The backfill was feeding a mechanism that
structurally could not rank.
Tiering now reads the RAW head magnitude, which genuinely lives on the
[1/3, 1] range these bounds were written for. Calibration keeps its real jobs -
AIConfidence() for MM sizing and SignedAIConfidence() for the vote are unchanged.
STILL OPEN, deliberately not touched here: the calibration TARGET itself.
empiricalAccuracy is 3-class agreement, which is the wrong quantity to scale a
DIRECTIONAL confidence against - it counts a Neutral class that is 0.19% of
labels. The honest target is the win rate on the calls the confidence describes
(directional precision), with the claimed-confidence average taken over those
same called bars. That needs a new accumulator and it interacts with the Neutral
over-calling being fixed elsewhere, so it wants one clean run first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 23:06:11 -04:00
//--- READS THE CALIBRATION BAND - never the window pass 3 grades. The deployed checkpoint is CHOSEN
//--- as the best-scoring era on the OOS window, so win rates measured back over that window are
//--- inflated by the selection, and these rows become filter weights: the selection set consumed
//--- twice, beside a deploy gate that applies a family-wise correction for exactly that effect.
//---
//--- The calibration band already has every property that needs (see the layout map on
//--- CalibPurgeBars): never trained on, never graded by pass 3 - which walks [0, oosCutoff) and so
//--- never reaches it - never seen by the deploy gate, and purged by a full label horizon on BOTH
//--- sides. A reserved slice carved out of the OOS window was tried for a few hours on 2026-08-16
//--- and removed: it bought the same property at the cost of 20% of the gate's sample, and in its
//--- first placement it also blanked ~10 months of chart arrows, because arrows are only ever drawn
//--- on bars pass 3 grades. This band costs the gate nothing and is larger besides.
//---
//--- One acknowledged impurity, and it is small: m_dirConfThreshold is FITTED on this band, and the
//--- walk applies that threshold when deciding which bars fired - so coverage here is mildly
//--- optimistic. That is one scalar fitted under a coverage floor, against checkpoint selection
//--- across hundreds of eras. Stated rather than hidden; see the completion log line.
int calibLo = CalibLoIndex ( oosCutoff ) ;
int calibHi = CalibHiIndex ( totalIter , oosCutoff ) ;
if ( CalibBandBars ( totalIter , oosCutoff ) < = 0 | | calibHi < = calibLo )
feat: derived taper restored; DB ranking reads a reserved slice, shrunk
TOPOLOGY - reverts the two constants and drops CausalHiddenLayerFloor.
The MQL5 article's 30%-per-layer cut and floor of 20 are load-bearing on ITS
first-layer width of 1000 (1000->300->90->27 needs a floor to stop). This
codebase MEASURES that width, and on the live SP500 H4 config it is 16 units -
already floored, with the budget printing "11360 estimated in-sample bars
cannot support a 800-wide input ... roughly 1.1 weights per training bar -
expect overfitting". At 16 units a floor of 20 makes lastHidden >=
m_initialNeuronsCount, so ComputeHiddenLayerCount returns on its first branch
and the width taper - the only part derived from this symbol's data - became
dead code on all four ensemble members, with depth (2 -> 4) set entirely by
counting feature domains. ComputeLayerWidths had already rejected this exact
pair of constants in its own comment.
The causal floor's premise does not hold either: layers are not inference
steps. The "1 layer linear / 2 nonlinear / 3 multi-connected" result is
Lippmann 1987 and is about hard-threshold units; with sigmoid/ReLU, Cybenko
1989 and Hornik 1991 give universal approximation from a single hidden layer.
Depth buys parameter efficiency for compositional functions, not reasoning
hops. ForceHiddenLayers remains for measuring depth directly.
RANKING SLICE - the backfill no longer reads the window it is judged on.
The deployed checkpoint is CHOSEN as the best-scoring era on the OOS window,
so win rates measured back over it are selection-inflated, and the backfill
was writing exactly those into the table filter weights rank on: the
selection set consumed twice, beside a deploy gate that applies a Sidak
correction for that effect. The newest RANK_SLICE_PCT_OF_OOS (20%) of the OOS
window, plus a label-horizon purge, is now reserved and graded by nothing -
not pass 3, not checkpoint selection, not the gate. The backfill reads only
that. The gate keeps ~80% of its measurement (power goes as the square root,
so ~10% of a sigma), and the slice is the newest data, which is the regime
about to be traded. RankSliceBars returns 0 when no honest slice fits and the
backfill then REFUSES and says so, rather than falling back to the scoring
window and looking like a success.
SHRINKAGE - per-tier win rates are shrunk toward the filter's own pooled rate
by MIN_TRADES_FOR_WIN_RATE pseudo-trades before becoming weights. The raw
ratio at the minimum sample count carries a ~15pp standard error, so a tier
that went 8-2 was handed weight 80 and outranked a tier measured over
hundreds of calls at 55 - the ranking was being driven by which small tier got
lucky. Opt-in per call site (priorWeight 0 keeps the raw behaviour).
Compile-verified: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 21:49:52 -04:00
{
m_dbBackfillDone = true ;
fix: drop the ranking slice for the calibration band; un-collapse the tiers
NOT COMPILED - user compiles.
(1) THE RANKING SLICE IS GONE. It reserved 20% of the OOS window so the
pattern-DB backfill would read bars the deployed checkpoint was not SELECTED on.
That objection stands; carving a new region to answer it did not. The calibration
band already has every property the slice was buying:
never trained on | never graded by pass 3 (which walks [0, oosCutoff) and so
never reaches it) | never seen by the deploy gate | purged by a full label
horizon on BOTH sides | and larger besides - 1,684 bars vs the ~970 carved
So the backfill now walks [calibLo, calibHi) and pass 3 goes back to grading the
entire OOS window, exactly as before any of this. The gate gets its full sample
back (~10% of a sigma), the split loses a region, and the failure mode found an
hour ago - a reserved region silently blanking ~10 months of chart arrows,
because arrows are only drawn on bars pass 3 grades - becomes impossible.
One impurity, stated in the completion log rather than hidden:
m_dirConfThreshold is FITTED on that band and the walk applies it to decide which
bars fired, so coverage there is mildly optimistic. One scalar under a coverage
floor, against checkpoint selection over hundreds of eras.
This backfill IS the deploy-time warm-up: it runs right after FinalizeTrainRun()
restores the deployed weights, so it scores with exactly what is about to trade.
(2) EVERY CALL WAS TIER 0, AND IT WAS ARITHMETIC. ConfidenceTier() quartiles
[floorConf, 1] where floorConf = 1/3 - the lowest magnitude a 3-way softmax
winner can hold. But it was fed CalibratedConfidenceMagnitude(), which multiplies
by m_confidenceCalScale, clamped to [0.3, 1.5]. That lower clamp is BELOW 1/3.
Whenever calibration bottoms out, t goes negative and MathMax(0, ...) pins every
call to tier 0.
Which is what the live run does. m_confidenceCalScale is EMA'd toward
empiricalAccuracy / avgClaimedConfidence; with the model over-calling Neutral,
3-class agreement sits near 10% against a claimed confidence near 0.9, so the
ratio is ~0.11 and clamps to 0.3 every era. Logged:
tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0)
828 calls, one bucket - the four tier weights and the entire per-tier pattern-DB
ranking reduced to a single number. The backfill was feeding a mechanism that
structurally could not rank.
Tiering now reads the RAW head magnitude, which genuinely lives on the
[1/3, 1] range these bounds were written for. Calibration keeps its real jobs -
AIConfidence() for MM sizing and SignedAIConfidence() for the vote are unchanged.
STILL OPEN, deliberately not touched here: the calibration TARGET itself.
empiricalAccuracy is 3-class agreement, which is the wrong quantity to scale a
DIRECTIONAL confidence against - it counts a Neutral class that is 0.19% of
labels. The honest target is the win rate on the calls the confidence describes
(directional precision), with the claimed-confidence average taken over those
same called bars. That needs a new accumulator and it interacts with the Neutral
over-calling being fixed elsewhere, so it wants one clean run first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 23:06:11 -04:00
Print ( ID + " : pattern-database backfill SKIPPED - this era carved no calibration band (study "
" window too short for OOS + two " + IntegerToString ( CalibPurgeBars ( ) ) + " -bar purges + a "
" band). Filter weights will build from real fills instead. Lengthen the study period or "
" lower the OOS split % to enable it. " ) ;
feat: derived taper restored; DB ranking reads a reserved slice, shrunk
TOPOLOGY - reverts the two constants and drops CausalHiddenLayerFloor.
The MQL5 article's 30%-per-layer cut and floor of 20 are load-bearing on ITS
first-layer width of 1000 (1000->300->90->27 needs a floor to stop). This
codebase MEASURES that width, and on the live SP500 H4 config it is 16 units -
already floored, with the budget printing "11360 estimated in-sample bars
cannot support a 800-wide input ... roughly 1.1 weights per training bar -
expect overfitting". At 16 units a floor of 20 makes lastHidden >=
m_initialNeuronsCount, so ComputeHiddenLayerCount returns on its first branch
and the width taper - the only part derived from this symbol's data - became
dead code on all four ensemble members, with depth (2 -> 4) set entirely by
counting feature domains. ComputeLayerWidths had already rejected this exact
pair of constants in its own comment.
The causal floor's premise does not hold either: layers are not inference
steps. The "1 layer linear / 2 nonlinear / 3 multi-connected" result is
Lippmann 1987 and is about hard-threshold units; with sigmoid/ReLU, Cybenko
1989 and Hornik 1991 give universal approximation from a single hidden layer.
Depth buys parameter efficiency for compositional functions, not reasoning
hops. ForceHiddenLayers remains for measuring depth directly.
RANKING SLICE - the backfill no longer reads the window it is judged on.
The deployed checkpoint is CHOSEN as the best-scoring era on the OOS window,
so win rates measured back over it are selection-inflated, and the backfill
was writing exactly those into the table filter weights rank on: the
selection set consumed twice, beside a deploy gate that applies a Sidak
correction for that effect. The newest RANK_SLICE_PCT_OF_OOS (20%) of the OOS
window, plus a label-horizon purge, is now reserved and graded by nothing -
not pass 3, not checkpoint selection, not the gate. The backfill reads only
that. The gate keeps ~80% of its measurement (power goes as the square root,
so ~10% of a sigma), and the slice is the newest data, which is the regime
about to be traded. RankSliceBars returns 0 when no honest slice fits and the
backfill then REFUSES and says so, rather than falling back to the scoring
window and looking like a success.
SHRINKAGE - per-tier win rates are shrunk toward the filter's own pooled rate
by MIN_TRADES_FOR_WIN_RATE pseudo-trades before becoming weights. The raw
ratio at the minimum sample count carries a ~15pp standard error, so a tier
that went 8-2 was handed weight 80 and outranked a tier measured over
hundreds of calls at 55 - the ranking was being driven by which small tier got
lucky. Opt-in per call site (priorWeight 0 keeps the raw behaviour).
Compile-verified: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 21:49:52 -04:00
return ;
}
fix: the DB backfill could never run, and HEAD did not compile
Four defects in 64c5dd5/1a05e63, found by review + a baseline compile.
Goals 1-8 of that session are unchanged; this makes 6 and 8 actually reachable.
1. HEAD DID NOT COMPILE - 6 errors. CControlPanel::Minimize/Maximize were
declared `virtual bool ... override`, but CAppDialog declares both as
`virtual void` (Controls\Dialog.mqh). errors 265 + 404 on each, plus 151
on `bool ok = CAppDialog::Minimize()`. Return type is void now; there was
never a success flag to forward. Verified: 0 errors, 0 warnings.
2. THE BACKFILL COULD NEVER ADVANCE, and neither could the OOS continual
simulation (that one has been dead since it was written). Both are armed
at the instant convergence is declared, and both advance only from inside
Train(), one chunk per call. But ScheduleTrainingIfNeeded's only per-tick
ArmStudyEvent site sits in the `else` of a branch taken whenever
m_trainingComplete is set and m_trainRunActive is clear - which is exactly
the state FinalizeTrainRun() leaves behind one line before they are armed.
Train() was never called again, so the walks sat at their start index
forever: no "simulation complete" line, and not one row written to the DB
this feature exists to fill. Only a manual Resume/Retrain unstuck them.
Both flags now keep the model schedulable.
3. IN AI_HYBRID - the mode this ships in - the backfill was never even armed.
Ensemble members deploy at Train() ENTRY and return immediately (so no era
is wasted), which skips the era-end block the backfill was started from.
All four members were a no-op for a second, independent reason. Armed on
the ensemble deploy path too, from m_resumeBars/m_resumeOosCutoff.
4. RE-RUNS DUPLICATED ROWS. RegisterSignal inserts unconditionally - no key,
no duplicate check - and m_dbBackfillDone is in-memory, so every later
attach that retrained to convergence wrote a second full set of rows for
the same bars. The ranking would count one bar once per model that ever
deployed, weighting superseded opinions as heavily as the live one. A
.dbfill marker stamps the deployed era; written only on completion (an
interrupted walk redoes itself rather than ranking a partial window) and
deleted with the other sidecars on reset-weights.
Also: WarmBlocking's timeout was silent, which restored the exact silent
pin failure it was added to prevent - it now says so in the journal, and
returns true for "no reference pairs to wait for" so the warning stays rare
enough to be read.
Not addressed, needs a decision: the backfill scores the OOS window with the
checkpoint that was SELECTED as best on that same window, then writes those
win rates into the table filter weights rank on - the selection set consumed
twice, undiscounted, while the deploy gate right next to it applies a
family-wise correction for exactly that effect. The rows are also simulated
triple-barrier outcomes at today's spread sharing a table with realised
fills. The completion log line now states both plainly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 21:25:51 -04:00
//--- ONE-SHOT ACROSS ATTACHES, not merely across this object's lifetime. m_dbBackfillDone is an
//--- in-memory flag, so every later attach that trains this configuration through to convergence
//--- again would walk the same OOS bars and write a second full set of rows - RegisterSignal()
//--- (Expert\ExpertSignalCustom.mqh) inserts unconditionally, with no key and no duplicate check.
//--- The ranking would then be counting the SAME bar several times, once per model that ever
//--- deployed here, weighting a superseded model's opinion exactly as heavily as the live one's.
//--- The marker stamps the era whose weights were used, so a redeploy of the same era is skipped
//--- and a genuinely retrained model (different era) is allowed through; reset-weights deletes it
//--- alongside the other sidecars (see LoadAndCompareTopologyConfiguration's discard block).
long deployedEra = ( m_ensembleMember & & g_ensBestEra > = 0 ) ? g_ensBestEra : ( long ) m_eraCount ;
int markerFlags = m_activeFileCommon ? FILE_COMMON : 0 ;
string markerFile = m_activeFileName + " .dbfill " ;
if ( FileIsExist ( markerFile , markerFlags ) )
{
//--- FILE_SHARE_READ|FILE_SHARE_WRITE on every open, without exception - a sibling chart holding
//--- this file open must not turn a skip-check into a hard failure (see the optimizer-cache
//--- corruption this rule came from).
int mh = FileOpen ( markerFile , markerFlags | FILE_TXT | FILE_READ | FILE_SHARE_READ | FILE_SHARE_WRITE ) ;
if ( mh ! = INVALID_HANDLE )
{
long stampedEra = StringToInteger ( FileReadString ( mh ) ) ;
FileClose ( mh ) ;
if ( stampedEra = = deployedEra )
{
m_dbBackfillDone = true ;
PrintVerbose ( ID + " : pattern-database backfill already done for era " +
IntegerToString ( ( int ) deployedEra ) + " - skipping (its rows are still in the DB). " ) ;
return ;
}
}
}
m_dbBackfillEra = deployedEra ;
2026-08-16 21:08:41 -04:00
dbm . OpenDatabase ( ) ;
//--- Frozen batch-norm statistics, exactly like pass 3's OOS scoring walk - see its comment for why
//--- an unfrozen forward pass would let the running stats drift while scoring.
Net . SetBatchNormFrozen ( true ) ;
m_dbBackfillBars = bars ;
fix: drop the ranking slice for the calibration band; un-collapse the tiers
NOT COMPILED - user compiles.
(1) THE RANKING SLICE IS GONE. It reserved 20% of the OOS window so the
pattern-DB backfill would read bars the deployed checkpoint was not SELECTED on.
That objection stands; carving a new region to answer it did not. The calibration
band already has every property the slice was buying:
never trained on | never graded by pass 3 (which walks [0, oosCutoff) and so
never reaches it) | never seen by the deploy gate | purged by a full label
horizon on BOTH sides | and larger besides - 1,684 bars vs the ~970 carved
So the backfill now walks [calibLo, calibHi) and pass 3 goes back to grading the
entire OOS window, exactly as before any of this. The gate gets its full sample
back (~10% of a sigma), the split loses a region, and the failure mode found an
hour ago - a reserved region silently blanking ~10 months of chart arrows,
because arrows are only drawn on bars pass 3 grades - becomes impossible.
One impurity, stated in the completion log rather than hidden:
m_dirConfThreshold is FITTED on that band and the walk applies it to decide which
bars fired, so coverage there is mildly optimistic. One scalar under a coverage
floor, against checkpoint selection over hundreds of eras.
This backfill IS the deploy-time warm-up: it runs right after FinalizeTrainRun()
restores the deployed weights, so it scores with exactly what is about to trade.
(2) EVERY CALL WAS TIER 0, AND IT WAS ARITHMETIC. ConfidenceTier() quartiles
[floorConf, 1] where floorConf = 1/3 - the lowest magnitude a 3-way softmax
winner can hold. But it was fed CalibratedConfidenceMagnitude(), which multiplies
by m_confidenceCalScale, clamped to [0.3, 1.5]. That lower clamp is BELOW 1/3.
Whenever calibration bottoms out, t goes negative and MathMax(0, ...) pins every
call to tier 0.
Which is what the live run does. m_confidenceCalScale is EMA'd toward
empiricalAccuracy / avgClaimedConfidence; with the model over-calling Neutral,
3-class agreement sits near 10% against a claimed confidence near 0.9, so the
ratio is ~0.11 and clamps to 0.3 every era. Logged:
tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0)
828 calls, one bucket - the four tier weights and the entire per-tier pattern-DB
ranking reduced to a single number. The backfill was feeding a mechanism that
structurally could not rank.
Tiering now reads the RAW head magnitude, which genuinely lives on the
[1/3, 1] range these bounds were written for. Calibration keeps its real jobs -
AIConfidence() for MM sizing and SignedAIConfidence() for the vote are unchanged.
STILL OPEN, deliberately not touched here: the calibration TARGET itself.
empiricalAccuracy is 3-class agreement, which is the wrong quantity to scale a
DIRECTIONAL confidence against - it counts a Neutral class that is 0.19% of
labels. The honest target is the win rate on the calls the confidence describes
(directional precision), with the claimed-confidence average taken over those
same called bars. That needs a new accumulator and it interacts with the Neutral
over-calling being fixed elsewhere, so it wants one clean run first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 23:06:11 -04:00
//--- Walks [calibLo, calibHi) from its OLDEST bar down to its newest - i.e. oldest -> newest in
//--- TIME, which is the order ProcessSignal's outdated-row guard requires. Both ends are clamped:
//--- the start against the feature-window bound, the stop against 2, so a degenerate band can only
//--- ever produce an empty walk, never one that wanders into the bars pass 3 grades.
m_dbBackfillStartIndex = ( int ) MathMin ( calibHi - 1 , bars - MathMax ( m_historyBars , 0 ) - 2 ) ;
m_dbBackfillStopIndex = ( int ) MathMax ( 2 , calibLo ) ;
2026-08-16 21:08:41 -04:00
m_dbBackfillIndex = m_dbBackfillStartIndex ;
m_dbBackfillFired = 0 ;
fix(gate): move the ranking slice to the OLD end - it walled off the recent chart
NOT COMPILED - user compiles.
User: "there is quite some trading going on, but absolutely nothing on the recent
area of the chart, like there is a hard wall starting around november 2025."
That wall is 7caf2f6's ranking slice, and it was placed at the wrong end. Chart
arrows are only ever drawn on bars pass 3 GRADES, and the slice reserved the
NEWEST 20% of the OOS window plus a label-horizon purge. At the live sizing -
~4,860 OOS bars, 128-bar horizon - that is ~1,100 H4 bars withheld from grading,
about ten months back from today, exactly where the wall appears.
The invisible cost was worse than the visible one: it handed the deploy gate the
OLDEST 80% of the OOS window and withheld the most recent regime from the single
decision that has to generalise forward.
Both fixed by putting the reserve at the oldest end instead:
[0, oosScoreHi) OOS - graded by pass 3 (NEWEST, arrows restored)
[oosScoreHi, rankLo) purge - one label horizon
[rankLo, oosCutoff) RANKING - backfill only, graded by nobody
[oosCutoff, calibLo) purge
[calibLo, calibHi) CALIBRATION
... IS
Of the three consumers competing for those bars, recency is worth least to the
ranking: it is an ORDERING of confidence tiers, far less regime-sensitive than an
absolute win rate, while the gate's power and the operator's read of the chart
both want the newest data. The slice keeps every property that made it worth
carving - never graded, never selected on, never seen by the gate, purged on both
sides - so the backfilled rows are still honestly out-of-sample.
RankSliceHiIndex is replaced by RankSliceLoIndex + OosScoreHiIndex; pass 3 now
excludes the slice at the TOP of its walk and descends to 2 as it always did.
The backfill walks [RankSliceLoIndex, oosCutoff) via a new m_dbBackfillStopIndex,
clamped at both ends so a degenerate slice yields an empty walk rather than one
that wanders into graded bars. Verified no reference to the old helper survives.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 22:58:55 -04:00
m_dbBackfillActive = ( m_dbBackfillStartIndex > = m_dbBackfillStopIndex ) ;
2026-08-16 21:08:41 -04:00
if ( ! m_dbBackfillActive )
m_dbBackfillDone = true ; // OOS window too short to walk - nothing to backfill, don't retry forever
}
//+------------------------------------------------------------------+
//| Time-boxed slice of the backfill walk - same chunking doctrine as |
//| every other long walk in this file (AdvanceOosSimulationChunk, |
//| AdvanceChartSignalRescan): a real forward pass per bar is genuine |
//| compute, so this yields on a wall-clock budget rather than running |
//| the whole OOS window in one call. Walks OLDEST -> NEWEST (mirrors |
//| pass 3's own m_oosScoreIndex descent) because ProcessSignal()'s |
//| outdated-row guard rejects a registration OLDER than a row its |
//| table already holds - inserting newest-first would have every |
//| older row rejected the instant the first one landed. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase : : AdvancePatternDatabaseBackfill ( void )
{
const uint DB_BACKFILL_TIME_BUDGET_MS = 80 ;
uint chunkStartTick = GetTickCount ( ) ;
dbm . BeginTransaction ( ) ;
//--- ConfidenceTier() reads the live dPrevSignal field (the panel/RefreshLatestSignal's source of
//--- truth) - borrowed per bar below to get the SAME tier bucketing a live vote would have used,
//--- then restored so this backfill walk never leaks into the live-facing signal.
double savedPrevSignal = dPrevSignal ;
fix(gate): move the ranking slice to the OLD end - it walled off the recent chart
NOT COMPILED - user compiles.
User: "there is quite some trading going on, but absolutely nothing on the recent
area of the chart, like there is a hard wall starting around november 2025."
That wall is 7caf2f6's ranking slice, and it was placed at the wrong end. Chart
arrows are only ever drawn on bars pass 3 GRADES, and the slice reserved the
NEWEST 20% of the OOS window plus a label-horizon purge. At the live sizing -
~4,860 OOS bars, 128-bar horizon - that is ~1,100 H4 bars withheld from grading,
about ten months back from today, exactly where the wall appears.
The invisible cost was worse than the visible one: it handed the deploy gate the
OLDEST 80% of the OOS window and withheld the most recent regime from the single
decision that has to generalise forward.
Both fixed by putting the reserve at the oldest end instead:
[0, oosScoreHi) OOS - graded by pass 3 (NEWEST, arrows restored)
[oosScoreHi, rankLo) purge - one label horizon
[rankLo, oosCutoff) RANKING - backfill only, graded by nobody
[oosCutoff, calibLo) purge
[calibLo, calibHi) CALIBRATION
... IS
Of the three consumers competing for those bars, recency is worth least to the
ranking: it is an ORDERING of confidence tiers, far less regime-sensitive than an
absolute win rate, while the gate's power and the operator's read of the chart
both want the newest data. The slice keeps every property that made it worth
carving - never graded, never selected on, never seen by the gate, purged on both
sides - so the backfilled rows are still honestly out-of-sample.
RankSliceHiIndex is replaced by RankSliceLoIndex + OosScoreHiIndex; pass 3 now
excludes the slice at the TOP of its walk and descends to 2 as it always did.
The backfill walks [RankSliceLoIndex, oosCutoff) via a new m_dbBackfillStopIndex,
clamped at both ends so a degenerate slice yields an empty walk rather than one
that wanders into graded bars. Verified no reference to the old helper survives.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 22:58:55 -04:00
for ( ; m_dbBackfillIndex > = m_dbBackfillStopIndex ; m_dbBackfillIndex - - )
2026-08-16 21:08:41 -04:00
{
if ( GetTickCount ( ) - chunkStartTick > = DB_BACKFILL_TIME_BUDGET_MS )
break ;
int oi = m_dbBackfillIndex ;
if ( ! ( oi < ( int ) ( m_dbBackfillBars - MathMax ( m_historyBars , 0 ) - 1 ) & &
oi < ArraySize ( m_labelCacheHasValue ) & & m_labelCacheHasValue [ oi ] ) )
continue ;
if ( ! BuildFeatureWindow ( oi ) | | ! Net . feedForward ( TempData ) )
continue ;
Net . getResults ( TempData ) ;
double oSignal = ( m_outputNeuronsCount = = 3 ) ? ApplyClassificationSoftmax ( ) : TempData [ 0 ] ;
double oDeploySignal = ( m_outputNeuronsCount = = 3 ) ? AdjustedSignalFromSoftmax ( ) : oSignal ;
ENUM_SIGNAL dir = DoubleToSignal ( oDeploySignal ) ;
if ( dir ! = Buy & & dir ! = Sell )
continue ; // Neutral/abstained - live voting would not have buffered a row for this bar either
dPrevSignal = oDeploySignal ;
int tier = ConfidenceTier ( ) ;
bool winLong = ( oi < ArraySize ( m_winLongCache ) ) ? m_winLongCache [ oi ] : false ;
bool winShort = ( oi < ArraySize ( m_winShortCache ) ) ? m_winShortCache [ oi ] : false ;
bool tradeWon = ( dir = = Buy ) ? winLong : winShort ;
double atr = m_ATR . Main ( oi ) ;
double closeAt = m_Close . GetData ( oi ) ;
if ( ! MathIsValidNumber ( atr ) | | atr < = 0.0 | | ! MathIsValidNumber ( closeAt ) | | closeAt < = 0.0 )
continue ;
//--- Same fill/exit convention TripleBarrierLabel() uses: long fills at close+spread and its
//--- target/stop are entry+reward/entry-risk; short fills at close and mirrors the two.
double spread = ( double ) m_symbol . Spread ( ) * m_symbol . Point ( ) ;
double slMult , tpMult ;
BarrierMultiples ( slMult , tpMult ) ;
double entryPrice , exitPrice ;
if ( dir = = Buy )
{
entryPrice = closeAt + spread ;
exitPrice = tradeWon ? entryPrice + tpMult * atr : entryPrice - slMult * atr ;
}
else
{
entryPrice = closeAt ;
exitPrice = tradeWon ? entryPrice - tpMult * atr : entryPrice + slMult * atr ;
}
MqlDateTime t ;
TimeToStruct ( m_Time . GetData ( oi ) , t ) ;
string pattern = " Pattern_ " + IntegerToString ( tier ) ;
string dirStr = ( dir = = Buy ) ? " Buy " : " Sell " ;
string tableName = PatternTableName ( GetFilterID ( ) , pattern , dirStr ) ;
double netVote = ( dir = = Buy ) ? PatternWeightForTier ( tier ) : - PatternWeightForTier ( tier ) ;
RegisterSignal ( t . year , t . mon , t . day , t . day_of_week , t . hour , t . min , tableName , pattern , dirStr ,
entryPrice , exitPrice , tradeWon ? " Profit " : " Loss " , netVote ) ;
m_dbBackfillFired + + ;
}
dPrevSignal = savedPrevSignal ;
dbm . CommitTransaction ( ) ;
fix(gate): move the ranking slice to the OLD end - it walled off the recent chart
NOT COMPILED - user compiles.
User: "there is quite some trading going on, but absolutely nothing on the recent
area of the chart, like there is a hard wall starting around november 2025."
That wall is 7caf2f6's ranking slice, and it was placed at the wrong end. Chart
arrows are only ever drawn on bars pass 3 GRADES, and the slice reserved the
NEWEST 20% of the OOS window plus a label-horizon purge. At the live sizing -
~4,860 OOS bars, 128-bar horizon - that is ~1,100 H4 bars withheld from grading,
about ten months back from today, exactly where the wall appears.
The invisible cost was worse than the visible one: it handed the deploy gate the
OLDEST 80% of the OOS window and withheld the most recent regime from the single
decision that has to generalise forward.
Both fixed by putting the reserve at the oldest end instead:
[0, oosScoreHi) OOS - graded by pass 3 (NEWEST, arrows restored)
[oosScoreHi, rankLo) purge - one label horizon
[rankLo, oosCutoff) RANKING - backfill only, graded by nobody
[oosCutoff, calibLo) purge
[calibLo, calibHi) CALIBRATION
... IS
Of the three consumers competing for those bars, recency is worth least to the
ranking: it is an ORDERING of confidence tiers, far less regime-sensitive than an
absolute win rate, while the gate's power and the operator's read of the chart
both want the newest data. The slice keeps every property that made it worth
carving - never graded, never selected on, never seen by the gate, purged on both
sides - so the backfilled rows are still honestly out-of-sample.
RankSliceHiIndex is replaced by RankSliceLoIndex + OosScoreHiIndex; pass 3 now
excludes the slice at the TOP of its walk and descends to 2 as it always did.
The backfill walks [RankSliceLoIndex, oosCutoff) via a new m_dbBackfillStopIndex,
clamped at both ends so a degenerate slice yields an empty walk rather than one
that wanders into graded bars. Verified no reference to the old helper survives.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 22:58:55 -04:00
if ( m_dbBackfillIndex > = m_dbBackfillStopIndex )
2026-08-16 21:08:41 -04:00
return ; // more slices to come
Net . SetBatchNormFrozen ( false ) ;
m_dbBackfillActive = false ;
m_dbBackfillDone = true ;
g_forcePatternWeightsRefresh = true ;
fix: the DB backfill could never run, and HEAD did not compile
Four defects in 64c5dd5/1a05e63, found by review + a baseline compile.
Goals 1-8 of that session are unchanged; this makes 6 and 8 actually reachable.
1. HEAD DID NOT COMPILE - 6 errors. CControlPanel::Minimize/Maximize were
declared `virtual bool ... override`, but CAppDialog declares both as
`virtual void` (Controls\Dialog.mqh). errors 265 + 404 on each, plus 151
on `bool ok = CAppDialog::Minimize()`. Return type is void now; there was
never a success flag to forward. Verified: 0 errors, 0 warnings.
2. THE BACKFILL COULD NEVER ADVANCE, and neither could the OOS continual
simulation (that one has been dead since it was written). Both are armed
at the instant convergence is declared, and both advance only from inside
Train(), one chunk per call. But ScheduleTrainingIfNeeded's only per-tick
ArmStudyEvent site sits in the `else` of a branch taken whenever
m_trainingComplete is set and m_trainRunActive is clear - which is exactly
the state FinalizeTrainRun() leaves behind one line before they are armed.
Train() was never called again, so the walks sat at their start index
forever: no "simulation complete" line, and not one row written to the DB
this feature exists to fill. Only a manual Resume/Retrain unstuck them.
Both flags now keep the model schedulable.
3. IN AI_HYBRID - the mode this ships in - the backfill was never even armed.
Ensemble members deploy at Train() ENTRY and return immediately (so no era
is wasted), which skips the era-end block the backfill was started from.
All four members were a no-op for a second, independent reason. Armed on
the ensemble deploy path too, from m_resumeBars/m_resumeOosCutoff.
4. RE-RUNS DUPLICATED ROWS. RegisterSignal inserts unconditionally - no key,
no duplicate check - and m_dbBackfillDone is in-memory, so every later
attach that retrained to convergence wrote a second full set of rows for
the same bars. The ranking would count one bar once per model that ever
deployed, weighting superseded opinions as heavily as the live one. A
.dbfill marker stamps the deployed era; written only on completion (an
interrupted walk redoes itself rather than ranking a partial window) and
deleted with the other sidecars on reset-weights.
Also: WarmBlocking's timeout was silent, which restored the exact silent
pin failure it was added to prevent - it now says so in the journal, and
returns true for "no reference pairs to wait for" so the warning stays rare
enough to be read.
Not addressed, needs a decision: the backfill scores the OOS window with the
checkpoint that was SELECTED as best on that same window, then writes those
win rates into the table filter weights rank on - the selection set consumed
twice, undiscounted, while the deploy gate right next to it applies a
family-wise correction for exactly that effect. The rows are also simulated
triple-barrier outcomes at today's spread sharing a table with realised
fills. The completion log line now states both plainly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 21:25:51 -04:00
//--- Stamp the marker only now, on completion: a walk interrupted half way (EA removed mid-chunk)
//--- leaves NO marker, so the next attach redoes it in full rather than ranking on a partial window.
//--- The duplicate rows that costs are the lesser error - a half-filled table is silently biased
//--- toward whichever end of the OOS window happened to finish.
{
int markerFlags = m_activeFileCommon ? FILE_COMMON : 0 ;
int mh = FileOpen ( m_activeFileName + " .dbfill " ,
markerFlags | FILE_TXT | FILE_WRITE | FILE_SHARE_READ | FILE_SHARE_WRITE ) ;
if ( mh ! = INVALID_HANDLE )
{
FileWriteString ( mh , IntegerToString ( ( int ) m_dbBackfillEra ) ) ;
FileClose ( mh ) ;
}
}
feat: derived taper restored; DB ranking reads a reserved slice, shrunk
TOPOLOGY - reverts the two constants and drops CausalHiddenLayerFloor.
The MQL5 article's 30%-per-layer cut and floor of 20 are load-bearing on ITS
first-layer width of 1000 (1000->300->90->27 needs a floor to stop). This
codebase MEASURES that width, and on the live SP500 H4 config it is 16 units -
already floored, with the budget printing "11360 estimated in-sample bars
cannot support a 800-wide input ... roughly 1.1 weights per training bar -
expect overfitting". At 16 units a floor of 20 makes lastHidden >=
m_initialNeuronsCount, so ComputeHiddenLayerCount returns on its first branch
and the width taper - the only part derived from this symbol's data - became
dead code on all four ensemble members, with depth (2 -> 4) set entirely by
counting feature domains. ComputeLayerWidths had already rejected this exact
pair of constants in its own comment.
The causal floor's premise does not hold either: layers are not inference
steps. The "1 layer linear / 2 nonlinear / 3 multi-connected" result is
Lippmann 1987 and is about hard-threshold units; with sigmoid/ReLU, Cybenko
1989 and Hornik 1991 give universal approximation from a single hidden layer.
Depth buys parameter efficiency for compositional functions, not reasoning
hops. ForceHiddenLayers remains for measuring depth directly.
RANKING SLICE - the backfill no longer reads the window it is judged on.
The deployed checkpoint is CHOSEN as the best-scoring era on the OOS window,
so win rates measured back over it are selection-inflated, and the backfill
was writing exactly those into the table filter weights rank on: the
selection set consumed twice, beside a deploy gate that applies a Sidak
correction for that effect. The newest RANK_SLICE_PCT_OF_OOS (20%) of the OOS
window, plus a label-horizon purge, is now reserved and graded by nothing -
not pass 3, not checkpoint selection, not the gate. The backfill reads only
that. The gate keeps ~80% of its measurement (power goes as the square root,
so ~10% of a sigma), and the slice is the newest data, which is the regime
about to be traded. RankSliceBars returns 0 when no honest slice fits and the
backfill then REFUSES and says so, rather than falling back to the scoring
window and looking like a success.
SHRINKAGE - per-tier win rates are shrunk toward the filter's own pooled rate
by MIN_TRADES_FOR_WIN_RATE pseudo-trades before becoming weights. The raw
ratio at the minimum sample count carries a ~15pp standard error, so a tier
that went 8-2 was handed weight 80 and outranked a tier measured over
hundreds of calls at 55 - the ranking was being driven by which small tier got
lucky. Opt-in per call site (priorWeight 0 keeps the raw behaviour).
Compile-verified: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 21:49:52 -04:00
Print ( ID + " : pattern database backfilled from " + IntegerToString ( m_dbBackfillFired ) + " calls on the "
fix: drop the ranking slice for the calibration band; un-collapse the tiers
NOT COMPILED - user compiles.
(1) THE RANKING SLICE IS GONE. It reserved 20% of the OOS window so the
pattern-DB backfill would read bars the deployed checkpoint was not SELECTED on.
That objection stands; carving a new region to answer it did not. The calibration
band already has every property the slice was buying:
never trained on | never graded by pass 3 (which walks [0, oosCutoff) and so
never reaches it) | never seen by the deploy gate | purged by a full label
horizon on BOTH sides | and larger besides - 1,684 bars vs the ~970 carved
So the backfill now walks [calibLo, calibHi) and pass 3 goes back to grading the
entire OOS window, exactly as before any of this. The gate gets its full sample
back (~10% of a sigma), the split loses a region, and the failure mode found an
hour ago - a reserved region silently blanking ~10 months of chart arrows,
because arrows are only drawn on bars pass 3 grades - becomes impossible.
One impurity, stated in the completion log rather than hidden:
m_dirConfThreshold is FITTED on that band and the walk applies it to decide which
bars fired, so coverage there is mildly optimistic. One scalar under a coverage
floor, against checkpoint selection over hundreds of eras.
This backfill IS the deploy-time warm-up: it runs right after FinalizeTrainRun()
restores the deployed weights, so it scores with exactly what is about to trade.
(2) EVERY CALL WAS TIER 0, AND IT WAS ARITHMETIC. ConfidenceTier() quartiles
[floorConf, 1] where floorConf = 1/3 - the lowest magnitude a 3-way softmax
winner can hold. But it was fed CalibratedConfidenceMagnitude(), which multiplies
by m_confidenceCalScale, clamped to [0.3, 1.5]. That lower clamp is BELOW 1/3.
Whenever calibration bottoms out, t goes negative and MathMax(0, ...) pins every
call to tier 0.
Which is what the live run does. m_confidenceCalScale is EMA'd toward
empiricalAccuracy / avgClaimedConfidence; with the model over-calling Neutral,
3-class agreement sits near 10% against a claimed confidence near 0.9, so the
ratio is ~0.11 and clamps to 0.3 every era. Logged:
tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0)
828 calls, one bucket - the four tier weights and the entire per-tier pattern-DB
ranking reduced to a single number. The backfill was feeding a mechanism that
structurally could not rank.
Tiering now reads the RAW head magnitude, which genuinely lives on the
[1/3, 1] range these bounds were written for. Calibration keeps its real jobs -
AIConfidence() for MM sizing and SignedAIConfidence() for the vote are unchanged.
STILL OPEN, deliberately not touched here: the calibration TARGET itself.
empiricalAccuracy is 3-class agreement, which is the wrong quantity to scale a
DIRECTIONAL confidence against - it counts a Neutral class that is 0.19% of
labels. The honest target is the win rate on the calls the confidence describes
(directional precision), with the claimed-confidence average taken over those
same called bars. That needs a new accumulator and it interacts with the Neutral
over-calling being fixed elsewhere, so it wants one clean run first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 23:06:11 -04:00
" held-out CALIBRATION band (bars " + IntegerToString ( m_dbBackfillStopIndex ) + " .. " +
IntegerToString ( m_dbBackfillStartIndex ) + " , era " + IntegerToString ( ( int ) m_dbBackfillEra ) +
" ) - this IS the deploy-time warm-up: it runs with the weights FinalizeTrainRun just restored, "
" so the per-pattern win-rate history describes exactly what is about to trade and no separate "
" backtest is needed first. Those bars were never trained on, never graded by pass 3 and never "
" seen by the deploy gate. Two honest caveats: they are SIMULATED triple-barrier outcomes at "
" today's spread rather than realised fills, and m_dirConfThreshold was fitted on this same "
" band, so coverage here is mildly optimistic. Small tiers are shrunk toward the pooled rate "
" before they become weights (see WinRateFromCounts). " ) ;
2026-08-16 21:08:41 -04:00
}
//+------------------------------------------------------------------+
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//| Alpha-balanced focal weight for one streamed bar - the cost-level |
//| imbalance correction used by BOTH the live continual-learning path |
//| and its OOS simulation. See the declaration comment and the |
//| ONLINE_LEARN_* block's CLASS IMBALANCE note for the derivation. |
//+------------------------------------------------------------------+
double CExpertSignalAIBase : : OnlineSampleWeight ( ENUM_SIGNAL trueSignal , double pBuy , double pSell , double pNeutral )
{
//--- Regression head has no class structure to balance.
if ( m_outputNeuronsCount ! = 3 )
return 1.0 ;
double weight = 1.0 ;
//--- alpha_c: inverse class frequency from the measured, persisted priors, normalised so the
//--- MAJORITY class is exactly 1.0 (a majority bar is never down-weighted below parity) and only
//--- minority bars are ever up-weighted. Unmeasured priors - a model deployed before any prior was
//--- recorded - skip the alpha term rather than divide by zero; focal's (1-p_t)^gamma still applies.
double priorMax = MathMax ( m_priorNeutral , MathMax ( m_priorBuy , m_priorSell ) ) ;
bool priorsUsable = ( priorMax > 0.0 & & m_priorBuy > 0.0 & & m_priorSell > 0.0 & & m_priorNeutral > 0.0 ) ;
if ( priorsUsable & & trueSignal ! = Neutral )
{
double truePrior = ( trueSignal = = Buy ) ? m_priorBuy : m_priorSell ;
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- Measured ratio scaled by ONLINE_LEARN_PARITY, then capped so a single rare bar can never
//--- deliver an outsized kick to an already-validated deployed model. Both were shared inputs
//--- until 2026-07-31; see the CLASS IMBALANCE note above ONLINE_LEARN_MAX_CLASS_WEIGHT for why
//--- this engine keeps its own cost-level correction now that Train() corrects in the gradient.
weight = MathMin ( MathMax ( 1.0 , ( priorMax / truePrior ) * ONLINE_LEARN_PARITY ) , ONLINE_LEARN_ALPHA_CAP ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- gamma: down-weights bars the model already gets right (the overwhelming Neutral majority), so
//--- the update concentrates on genuinely informative confirmations. Constant since 2026-07-31.
if ( ONLINE_LEARN_FOCAL_GAMMA > 0.0 )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
double pt = ( trueSignal = = Buy ) ? pBuy : ( trueSignal = = Sell ) ? pSell : pNeutral ;
pt = MathMax ( 0.0 , MathMin ( 1.0 , pt ) ) ;
refactor(ai): nine class-imbalance inputs down to two
The imbalance section offered nine controls for one job. Audited against the
code, five of them did not do what their names said at the shipped defaults:
AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns
whenever the adjusted loss is on, which is default.
OversampleParity DEAD in training - Training.mqh gated the replay loop
on !useLogitAdjustedLoss (correctly, citing Buda et
al. 2018). Live only in the online-learning path.
EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma
damper - "replay minority bars through pass-2
oversampling" was a focal-loss switch.
ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25.
UseStaticPrior An exact duplicate of FreezePriorCalibration - the two
were OR'd together in the single place either is read.
So they were not five mechanisms fighting; they were one mechanism plus eight
knobs that mostly described machinery that no longer ran. That is worse than
a real conflict, because the log agreed with the names: the label-cache line
printed "reps up to 28x (90% parity) (seeding era 0's class-balance
oversampling)" on every run, describing an oversampling pass that had been
switched off. It is fixed here too - it cost this session a wrong diagnosis.
The one genuine redundancy was focal loss, running at gamma*0.125 alongside
the adjusted loss: two corrections on the same axis, the exact stacking
failure this file already cited Buda et al. for in two other places, damped
by a replay flag whose replay path was itself dead. Removed rather than
re-tuned. The plateau ladder is unaffected - its escape is the learning-rate
warm restart; the gamma anneal beside it only ever stepped toward zero.
WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze:
LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted-
Loss boolean, since a strength dial where 0 already
means off does not need an on/off switch beside it.
FreezePriorCalibration unchanged.
It is the only one of the six corrections with a consistency guarantee, and
it is consistent for exactly the balanced-error metric checkpoint selection
already ranks on - so the loss and the deploy decision optimize one thing.
The online continual-learning path keeps its own alpha-balanced focal weight,
now as constants pinned to the removed inputs' shipped defaults, so its
behaviour is unchanged. It legitimately needs its own correction:
ApplyLogitAdjustment() only runs inside a training run, so a deployed model
that was reloaded carries no logit offsets and would otherwise stream 31:1
data into itself uncorrected.
The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a
double fed to a %d conversion and had always emitted a literal 0; the |MR:
segment is written as the constant its shipped defaults produced. Dropping
either would have re-keyed every model and forced a from-scratch retrain of
the one topology currently converged and trading.
Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS,
OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable
"neutralized by prior correction" diagnostic.
Both builds compile 0 errors, 0 warnings. No retrain forced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
weight * = MathPow ( 1.0 - pt , ONLINE_LEARN_FOCAL_GAMMA ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
return weight ;
}
//+------------------------------------------------------------------+
//| Online continual-learning step - LIVE CHART ONLY. Once a model is |
//| deployed (m_trainingComplete) it keeps learning from real market |
//| structure the same supervised way it was trained: predicting the |
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//| TRIPLE-BARRIER outcome for each bar. The critical rule the user |
//| asked for is the confirmation delay - a bar's barrier label is not |
//| knowable until m_barrierHorizonBars more bars have closed after it |
//| (that is the vertical barrier itself), so the model must NEVER |
//| backprop the newest bars against an unresolved outcome, even |
//| though it happily EMITS a live signal on them. This method |
//| therefore only ever learns from the "confirmable frontier" and |
//| older: the newest bar whose now-relative index is |
//| >= m_barrierHorizonBars. Everything newer than that is inference- |
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//| only until it, too, matures - identical to how training holds its |
//| recent bars in the OOS holdout and embargoes the boundary band. |
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//| (Was m_swingConfirmationBars, which answered the ZigZag repainting |
//| question. That is no longer the label's lookahead - see |
//| m_barrierHorizonBars.) |
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//| |
//| Mechanism per newly-matured bar (oldest->newest, exactly |
//| AdvanceOosSimulationChunk()'s predict-score-then-learn step, but |
//| on the REAL deployed Net): build the same feature window training |
//| used, feedForward, score the prediction against the confirmed |
//| label (guardrail EMA), then backProp that label. The deployed |
//| SHADOW is nudged toward Net by SHADOW_WEIGHT_TAU only while the |
//| rolling accuracy holds up; if it decays the blend FREEZES (live |
//| keeps trading the last-good shadow, Net keeps adapting so it can |
//| recover) - drift can never reach the account. State persists in the |
//| .stats sidecar so a restart neither re-learns old bars nor skips. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase : : OnlineLearnStep ( void )
{
//--- Hard gates. m_inferenceOnly covers BOTH the single backtest and every optimization pass (see
//--- its declaration comment): in the tester the model is held FIXED, so continual learning is a
//--- live-chart-only behaviour (forward-test it on a demo account, not the Strategy Tester).
perf(autotune): replace the genetic search with a filter score - hours to seconds
MEASURED COST OF THE GA, which is what retired it. Per generation:
rung 0: 8 cand x 3 seeds x 3 eras = 72 eras
rung 1: 4 cand x 3 seeds x 8 eras = 96
rung 2: 2 cand x 3 seeds x 20 eras = 120
= 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's
real training began. Against the observed era times on SP500 H1:
PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22)
CONV 41.3 s/era -> 13.2 h
LSTM 150.4 s/era -> 48.1 h
HYBRID 154.6 s/era -> 49.5 h
Two days to tune is not a first-run experience, and it is the phase in
which the panel goes quiet, which is what made it look like a hang.
It also bought nothing. The space is 90 points (10 MA periods x 9 MA
types), so 1152 evaluations revisited each point ~13 times; and rungs of
3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run
proves it: every finalist scored 25.0-25.9% balanced accuracy - below the
33.3% one-class floor, i.e. indistinguishable noise - and the search then
"deployed the winner" of that.
THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full
training run to choose a feature's period is a wrapper method paying
wrapper prices for a decision that does not need one. The reference book
does not do this: ch. 3.3 selects inputs by measuring each candidate
indicator's CORRELATION with the target and dropping the ones with none,
with no network involved.
So: rank candidates by the MUTUAL INFORMATION between the resulting
feature vector and the triple-barrier label. MI rather than correlation
because the label is 3-class categorical and the features are not
monotonically related to it. Equal-FREQUENCY binning (rank-based),
because these features are ATR-normalised and heavy-tailed - fixed-width
bins put nearly everything in one bucket and report ~0 information for a
genuinely useful feature.
Scoring is arithmetic over the feature cache, so it costs seconds and its
cost is independent of topology: LSTM now tunes as fast as the MLP.
Coordinate sweep, not product sweep - cost is the SUM of per-parameter
candidate counts, so enabling every indicator stays affordable - with a
second pass that breaks early once nothing moves.
Sampling is IS-ONLY. Letting the OOS window influence which indicator
settings ship would mean the holdout had been used for selection and had
stopped being a holdout.
HONEST LIMIT, recorded because it is the price: MI is marginal, so a
parameter that only pays off in combination with another can be missed
(Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it
replaces was ranking pure noise at 48 h a run, this is strictly better.
Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/
GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga*
members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget.
AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28
read sites all permanently inert. That is not a tidy-up: the `if
(!m_evalMode)` guard on UpdateClassPriors is exactly what silently
disabled the imbalance correction for entire runs two commits ago. Dead
machinery that still reads like live machinery is this codebase's most
expensive recurring bug, and leaving 28 more instances of it would have
been indefensible.
The panel's tuning-progress state goes too - tuning no longer takes long
enough to need one.
Both builds compile 0 errors / 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
if ( ! m_enableOnlineLearning | | m_inferenceOnly | | m_trainRunActive )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
return ;
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus
The NN now has a target that is not per-bar direction (closed, best-of-999
p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of
cost). One net for all 52 pattern-sides, AIType=AI_META.
- NetForward.mqh: the host-side softmax+CE gradient generalized total==3 ->
2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no
compute backend changes.
- SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on
disk (decoupled from the config fingerprint that burned four S1 runs); the
GMT->server offset is measured PER ROW against entryPrice vs bar open
(DST-immune, histogram logged); a window-span regime filter drops the
pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the
input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR).
- Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate
calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell
lets checkpoint selection, the edge floor, the plateau ladder and the
family-wise deploy gate run UNCHANGED: precision reads as win rate among
traded candidates, chance as the base win rate, recalls as sensitivity/
specificity. Era-end META line: coverage x (p - break-even) vs the null.
- Labels are the side-conditional triple-barrier win caches - never the DB's
stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base
rate). Live inference + online learning guarded off until S3.
- Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename
slot keep meta models fully separate from direction models.
Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with
AIType=AI_META; S3 wires the votes via the per-side hooks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- Meta target: online continual learning is direction-shaped (3-slot targets, bar labels) and
//--- the meta head's live path does not exist until S3 - hold the model fixed.
if ( IsMetaTarget ( ) )
return ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
if ( ! m_trainingComplete | | m_trainingStopRequested | | m_trainingPaused )
return ;
if ( MQLInfoInteger ( MQL_TESTER ) | | MQLInfoInteger ( MQL_OPTIMIZATION ) | | MQLInfoInteger ( MQL_FORWARD ) )
return ; // belt-and-braces: never adapt weights inside any tester context
if ( CheckPointer ( Net ) = = POINTER_INVALID | | Net . CpuInference ( ) )
return ; // DLL-free inference build has no backend to backprop through
if ( CheckPointer ( m_shadowNet ) = = POINTER_INVALID )
return ; // nothing deployed to blend into yet (RefreshLatestSignal bootstraps it first)
if ( m_outputNeuronsCount ! = 1 & & m_outputNeuronsCount ! = 3 )
return ;
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
int conf = MathMax ( m_barrierHorizonBars , 1 ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
int barsAvail = Bars ( m_symbol . Name ( ) , PERIOD_CURRENT ) ;
//--- Need the frontier bar (now-relative index conf) plus a full feature window BEHIND it, plus a
//--- little slack so a short catch-up walk stays in-bounds.
int need = conf + ( int ) m_historyBars + 2 ;
if ( barsAvail < need )
return ;
//--- Load enough history for the frontier window and a bounded catch-up; RefreshConvergedSignal()
//--- only sized buffers relative to dtStudied (newest bars), which is too shallow to reach the
//--- confirmation frontier.
int wantBars = MathMin ( need + ONLINE_LEARN_MAX_CATCHUP , barsAvail ) ;
fix(depth): route EVERY ResizeBuffers call site through one indicator-depth gate
1dda479 clamped the training sweep. It left five other paths asking the indicators
for a depth they cannot serve, and on a live account the quiet ones are worse than
the stall was - a stalled chart is visible, a chart trading on a degraded feature
window is not.
ServableBars(want, context) is now the single gate, and all six go through it:
training sweep clamp, floored at TRAIN_MIN_CLAMPED_BARS (below that a small
positive BarsCalculated is warm-up, which m_coldSweepTick owns)
label prebuild clamp - labels come from price/ADZigZag and would survive a
capped MA, but ResizeBuffers sizes EVERY buffer and a failed
CopyBuffer leaves m_MA EMPTY for the next reader, so this path
could silently re-break the block Train()'s clamp just fixed
live inference HOLD. Below `need` the swing block takes its degraded path and
inference runs on a different feature distribution than the model
was fitted on. This EA sizes real positions off that output, so
no signal beats a mismatched one
online learning HOLD, same reason and worse - this path WRITES to a live trading
model, so a mismatched (features, label) pair is not a wrong arrow,
it is a wrong weight update that compounds every bar
chart rescan clamp - SIGNAL_RESCAN_LOOKBACK_BARS is 5000 and MT5's smallest
"Max bars in chart" is also 5000, so this one is genuinely
reachable; uncapped it repaints the window all-Neutral
research export clamp before the emptiness test, so a capped symbol exports the
depth it has rather than writing a CSV with a dead feature block -
an artefact that looks complete and is silently wrong
Both HOLDs are insurance, not expected states: `need` tops out near 1,152 bars
(16 + 750 + 384 + 2) against a 5,000 floor on the terminal setting. They exist so
the failure mode is unreachable rather than merely unlikely.
Not changed: a genuinely SHORT price history still takes the old degraded path at
every site. That is pre-existing behaviour and narrowing it would mute charts that
trade today, so it stays a separate decision rather than a side effect of this fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:33:16 -04:00
//--- HOLD RATHER THAN LEARN ON A SHORT WINDOW. `need` is the minimum that reaches the confirmation
//--- frontier WITH a full feature window behind it; below it the swing block silently degrades and
//--- the features stop matching the ones the model was fitted on. Everywhere else that is a wrong
//--- arrow - here it is a wrong WEIGHT UPDATE applied to a live, trading model, and it compounds
//--- every bar. The catch-up slack on top of `need` is optional and may be clamped away; `need`
//--- itself is not, so a cap that eats into it stops the update instead of corrupting it.
int servable = ServableBars ( wantBars , " online learning " ) ;
if ( servable < need )
return ;
wantBars = servable ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
if ( ! ResizeBuffers ( wantBars ) | | ! RefreshData ( ) )
return ;
fix: the sequence models were reading the window backwards
BuildFeatureWindow() replaces eight hand-rolled copies of the same loop
and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first,
because MQL5 timeseries indices run backwards and `r + b` with b ascending
walks into the past.
Harmless for PAI and CONV - a dense layer learns a weight per position
either way, a conv learns time-mirrored kernels. Not harmless for the
recurrent stacks:
- LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t.
- It writes output[] only when t == steps-1: the visible output IS the
last hidden state.
- c_t = f*c_{t-1} + i*g decays toward the start of the sequence.
lstm_seq_flowcheck.cpp measured block 0's influence on the output at
1.2e-2 of block T-1's, at the shipped forget bias of 1.0.
So the bar being PREDICTED sat at the far end of the decay and the output
was handed to the OLDEST bar in the window - the exact inverse of what the
window is for. ~80x backwards on LSTM and HYBRID, on all three tiers
(OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never
surfaced as a backend discrepancy.
This does not create edge - the MI diagnostics read at the noise floor
(p=0.4975) with a working positive control. It makes the one hypothesis
those diagnostics explicitly do NOT cover testable: they are marginal and
per-bar, and state they "cannot rule out one that only exists in
combination or across time". The sequence model is the instrument for
across-time structure and it has been crippled, so that hypothesis has
never been honestly tested.
Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and
its features, so a stale .nnw would load cleanly and run a model fitted to
one ordering against the other, silently. Re-keying every config is the
point, not collateral damage. FORCES A FULL RETRAIN.
Also: the now-relative bar caches are re-keyed on the two live paths.
EnsureBarCachesCapacity() was only ever called from training paths, but
once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar
to RefreshConvergedSignal() and Train() is never re-entered - so nothing
cleared the feature cache again for the life of the process. A chart that
trained to convergence kept replaying the rows computed for the last
training era's bar grid: the live signal froze at its convergence-time
value, and OnlineLearnStep() backpropped those stale features against
freshly resolved labels. Backtests were never affected (an inference-only
process never allocates the arrays, so every read recomputes).
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 18:28:44 -04:00
//--- Same now-relative invalidation RefreshConvergedSignal() does, and needed independently of it: this
//--- runs on a DEEPER bar grid (wantBars reaches the confirmation frontier, that one only reaches the
//--- newest feature window), so the two legitimately disagree about `bars` and each must re-key the
//--- cache for the grid it is about to read. Stale rows matter more here than anywhere else - this is
//--- the one path that WRITES to a live, trading model, so a mismatched (features, label) pair is not a
//--- wrong arrow, it is a wrong weight update. See RefreshConvergedSignal()'s comment for why nothing
//--- else clears this once training has completed.
//--- Note the catch-up walk below is unaffected in cost: this fires once per call, before the loop, so
//--- the overlapping windows inside the loop still share cached rows.
EnsureBarCachesCapacity ( wantBars ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
datetime frontierTime = m_Time . GetData ( conf ) ;
if ( frontierTime < = 0 )
return ;
//--- First step of this deployment (or a model that never online-learned): DON'T retroactively
//--- backfill the whole history through backprop in one shot - that could shift the just-validated
//--- deployed model materially before any live confirmation. Anchor the watermark at the current
//--- frontier and begin learning from genuinely new confirmations forward.
if ( m_onlineLearnedUpToTime < = 0 )
{
m_onlineLearnedUpToTime = frontierTime ;
return ;
}
if ( frontierTime < = m_onlineLearnedUpToTime )
return ; // no bar has matured past the watermark since last time
//--- Seed the guardrail EMA from the model's deploy-time OOS accuracy the first time we actually
//--- learn, so the floor is meaningful from the very first update (not a cold 0 that would trip it).
if ( m_onlineRollingAcc < 0.0 )
m_onlineRollingAcc = ( dForecast > 0.0 & & dForecast < = 100.0 ) ? dForecast : 100.0 ;
//--- Guardrail floor: deploy baseline minus a margin, never below the absolute minimum.
double baseline = ( dForecast > 0.0 & & dForecast < = 100.0 ) ? dForecast : 100.0 ;
double accFloor = MathMax ( ONLINE_LEARN_MIN_ACC , baseline - ONLINE_LEARN_ACC_MARGIN ) ;
//--- Find the oldest not-yet-learned confirmed bar: walk from the frontier (index conf) toward older
//--- bars (increasing index) until we pass the watermark or hit the catch-up cap, then learn newest-
//--- ward from there so bars are consumed in strict chronological (oldest->newest) order.
int oldestIdx = conf ;
while ( oldestIdx < barsAvail - 1
& & oldestIdx < conf + ONLINE_LEARN_MAX_CATCHUP
& & m_Time . GetData ( oldestIdx ) > m_onlineLearnedUpToTime )
oldestIdx + + ;
//--- oldestIdx now points at the first bar whose time is <= watermark (already learned) or the cap;
//--- the newest UNLEARNED bar is one step newer (idx-1). Learn from idx = oldestIdx-1 down to conf.
//--- Pin the learning rate for the duration of this walk and restore it after: `eta` is a GLOBAL
//--- shared by every signal instance, so leaving it modified would corrupt another model's training
//--- chunk - see ONLINE_LEARN_ETA_SCALE's comment.
double savedEta = eta ;
eta = m_modelEta * ONLINE_LEARN_ETA_SCALE ;
int learned = 0 ;
for ( int idx = oldestIdx - 1 ; idx > = conf ; idx - - )
{
datetime bt = m_Time . GetData ( idx ) ;
if ( bt < = m_onlineLearnedUpToTime )
continue ; // already learned (defensive; the walk above should exclude it)
//--- Build this bar's feature window - IDENTICAL to Train()/RefreshLatestSignal(): ends AT bar idx
//--- and extends m_historyBars into the past. No lookahead (all bars are older than idx).
fix: the sequence models were reading the window backwards
BuildFeatureWindow() replaces eight hand-rolled copies of the same loop
and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first,
because MQL5 timeseries indices run backwards and `r + b` with b ascending
walks into the past.
Harmless for PAI and CONV - a dense layer learns a weight per position
either way, a conv learns time-mirrored kernels. Not harmless for the
recurrent stacks:
- LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t.
- It writes output[] only when t == steps-1: the visible output IS the
last hidden state.
- c_t = f*c_{t-1} + i*g decays toward the start of the sequence.
lstm_seq_flowcheck.cpp measured block 0's influence on the output at
1.2e-2 of block T-1's, at the shipped forget bias of 1.0.
So the bar being PREDICTED sat at the far end of the decay and the output
was handed to the OLDEST bar in the window - the exact inverse of what the
window is for. ~80x backwards on LSTM and HYBRID, on all three tiers
(OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never
surfaced as a backend discrepancy.
This does not create edge - the MI diagnostics read at the noise floor
(p=0.4975) with a working positive control. It makes the one hypothesis
those diagnostics explicitly do NOT cover testable: they are marginal and
per-bar, and state they "cannot rule out one that only exists in
combination or across time". The sequence model is the instrument for
across-time structure and it has been crippled, so that hypothesis has
never been honestly tested.
Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and
its features, so a stale .nnw would load cleanly and run a model fitted to
one ordering against the other, silently. Re-keying every config is the
point, not collateral damage. FORCES A FULL RETRAIN.
Also: the now-relative bar caches are re-keyed on the two live paths.
EnsureBarCachesCapacity() was only ever called from training paths, but
once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar
to RefreshConvergedSignal() and Train() is never re-entered - so nothing
cleared the feature cache again for the life of the process. A chart that
trained to convergence kept replaying the rows computed for the last
training era's bar grid: the live signal froze at its convergence-time
value, and OnlineLearnStep() backpropped those stale features against
freshly resolved labels. Backtests were never affected (an inference-only
process never allocates the arrays, so every read recomputes).
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 18:28:44 -04:00
//--- "Identical" is now enforced rather than asserted - all three go through BuildFeatureWindow().
if ( ! BuildFeatureWindow ( idx ) )
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
//--- window not buildable this bar (e.g. an indicator hole) - advance the watermark past it so
//--- we don't wedge re-trying the same bar forever, but learn nothing from it.
m_onlineLearnedUpToTime = bt ;
continue ;
}
//--- Predict with the CURRENT (pre-update) weights, then score against the confirmed label for the
//--- rolling guardrail - exactly AdvanceOosSimulationChunk()'s predict-before-learn measurement.
Net . feedForward ( TempData ) ;
Net . getResults ( TempData ) ;
double predSignal = ( m_outputNeuronsCount = = 3 ) ? ApplyClassificationSoftmax ( ) : TempData [ 0 ] ;
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property
of the market. Labelling only the exact bar where a ZigZag pivot confirms
gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism
this codebase accumulated sits downstream of that one choice: the
logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias
seed, balanced-accuracy-then-precision selection with its coverage floor,
the recall floor and its catch-22, the alternation gate, NMS, and the four
oversampling designs that collapsed before them.
The reference this engine is built on (references/neuronetworksbook.pdf
ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT
EXTREMUM on every bar - ~50/50 by construction, with no imbalance to
correct at all. It never had this problem because it never asked "is this
the pivot bar".
Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's
OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its
target before its stop, within a horizon. Buy = long resolves, Sell =
short resolves, Neutral = neither. Consequences:
- dir-precision in the era line stops being a proxy and becomes the win
rate of the strategy under its own exit rules.
- Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e.
~2:1 instead of 31:1. Measured and logged at the end of the prebuild.
- Spread is charged on both legs, so it is a NET win rate.
- Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches
inside one bar and the optimistic reading is how a backtested edge
becomes a live loss.
ZigZag stays as input features (EnableSwingContext) and now also supplies
the vertical barrier: the horizon is the median confirmed leg length,
snapped to a coarse ladder. Derived, not configured, and deliberately kept
out of the filename fingerprint - a filename keyed on a measured quantity
orphans a trained model the moment the measurement moves.
Removed, because the premise died with the old target:
- the alternation gate. Correct for pivot labels (a ZigZag cannot emit two
same-type pivots in a row, so a repeat was provably a false fire), and
wrong for barrier labels, which answer each bar independently. It also
took its worst consequence with it: a one-sided model previously got ONE
trade per backtest, a hard blocker on marketplace validation.
- SignalClusterWindow now defaults off - it de-duplicated repeats that are
now real trades. Kept as an opt-in display control.
- LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel.
- the era-0 output-bias seed now needs a genuinely dominant class (0.70)
rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a
correction.
Also fixed, both found while wiring the above:
1. RefreshConvergedSignal sized its buffers from a date delta
(Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training
watermark; in the tester it is loaded from a live-chart save AHEAD of
the simulated date, so the interval inverted, Bars() returned ~0, and
the buffer came out at exactly m_historyBars - deep enough for the OHLC
window and far too shallow for the Donchian-50 / 20-bar-return / SMA
extension behind it. Inference silently computed DIFFERENT features
from the ones training learned on, live as well as in the tester. Now
sized from what the feature builder actually needs.
2. The barrier horizon is resolved on the deployed path too. A deployed
model never enters Train(), so it never reached the prebuild, and
OnlineLearnStep reads the horizon as its confirmation delay - left at
the fallback it would have backpropped bars whose barriers had not
resolved. Silent lookahead in the one place that writes to a live model.
SL_Mode/TP_Mode join the weights fingerprint: they define the labels now,
so a model trained at 1:3 must never be silently reused at 1:1. This
re-keys every pre-existing model by design - none were trained on this task.
Inference census extended with the vote gate. LongCondition/ShortCondition
open with a readiness check the refresh counters never see; in the tester it
reduces to "the seeded _optcache.nnw must have LOADED", and if it did not,
every vote is hard-zeroed while the model still answers Buy. The old three
counters would have read that as "the model says Neutral" - false, and a
completely different fix. This is the leading candidate for the
zero-direction backtest and the census can now name it in one run.
Both builds compile 0 errors / 0 warnings. Forces a full retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- Same target rule training used. `idx` is at or beyond the confirmation frontier (conf ==
//--- m_barrierHorizonBars, enforced above), so the forward window this reads is fully closed.
ENUM_SIGNAL trueSignal = TripleBarrierLabel ( idx ) ;
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
bool hit = ( DoubleToSignal ( predSignal ) = = trueSignal ) ;
m_onlineRollingAcc + = ( 100.0 * ( hit ? 1.0 : 0.0 ) - m_onlineRollingAcc ) / ONLINE_ACC_SMOOTH ;
//--- Per-class softmax probabilities as of THIS bar's pre-update prediction. Must be read here,
//--- before TempData is rebuilt as the target vector below - ApplyClassificationSoftmax() has
//--- already normalised TempData[0..2] in place into a genuine distribution (same contract pass 2
//--- relies on for its own focal term).
double pBuy = ( TempData . Total ( ) > 0 ) ? TempData . At ( 0 ) : 0.0 ;
double pSell = ( TempData . Total ( ) > 1 ) ? TempData . At ( 1 ) : 0.0 ;
double pNeutral = ( TempData . Total ( ) > 2 ) ? TempData . At ( 2 ) : 0.0 ;
//--- Build the target vector - identical encoding to Train()/AdvanceOosSimulationChunk().
bool buy = ( trueSignal = = Buy ) ;
bool sell = ( trueSignal = = Sell ) ;
TempData . Clear ( ) ;
if ( m_outputNeuronsCount = = 1 )
TempData . Add ( buy & & ! sell ? 1 : ( ! buy & & sell ? -1 : 0 ) ) ;
else
{
TempData . Add ( buy ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW ) ;
TempData . Add ( sell ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW ) ;
TempData . Add ( ( ! buy & & ! sell ) ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW ) ;
}
Net . backProp ( TempData , OnlineSampleWeight ( trueSignal , pBuy , pSell , pNeutral ) ) ;
m_onlineSamples + + ;
//--- Deploy the improvement ONLY while accuracy holds. Warmup: allow the first few blends (the
//--- model was just validated at deploy, steps are tiny) until the EMA has enough samples to judge.
bool blendOk = ( m_onlineSamples < = ONLINE_LEARN_WARMUP ) | | ( m_onlineRollingAcc > = accFloor ) ;
if ( blendOk )
{
m_shadowNet . BlendWeightsFrom ( Net , SHADOW_WEIGHT_TAU ) ;
if ( m_onlineBlendFrozen )
{
m_onlineBlendFrozen = false ;
Print ( ID + " : online-learning deployment RESUMED - rolling accuracy recovered to "
+ DoubleToString ( m_onlineRollingAcc , 1 ) + " % (floor " + DoubleToString ( accFloor , 1 ) + " %) " ) ;
}
}
else if ( ! m_onlineBlendFrozen )
{
m_onlineBlendFrozen = true ;
Print ( ID + " : online-learning deployment FROZEN - rolling accuracy " + DoubleToString ( m_onlineRollingAcc , 1 )
+ " % fell below floor " + DoubleToString ( accFloor , 1 ) + " %; live keeps trading the last-good model while it adapts " ) ;
}
m_onlineLearnedUpToTime = bt ;
learned + + ;
m_onlineBarsSincePersist + + ;
}
//--- Hand the shared global back exactly as found, on BOTH exit paths below - see the matching
//--- savedEta assignment above for why this must not leak out of this function.
eta = savedEta ;
if ( learned < = 0 )
return ;
//--- Periodic durable persistence so a crash loses at most ONLINE_LEARN_PERSIST_EVERY bars of
//--- adaptation (shutdown also persists via PersistOnShutdown()).
if ( m_onlineBarsSincePersist > = ONLINE_LEARN_PERSIST_EVERY )
{
double ip [ ] ;
m_indicatorTuner . Flatten ( ip ) ;
bool saveOk = Net . Save ( m_activeFileName + " .nnw " , dError , dUndefine , dForecast , dtStudied , m_activeFileCommon , m_eraCount , m_trainingComplete , ip ) ;
if ( ! saveOk )
Print ( ID + " : ERROR - online-learning Net.Save failed for " + m_activeFileName + " .nnw. Retrying next persist interval instead of resetting the bars-since-persist counter. " ) ;
SaveShadowNet ( ip ) ;
if ( ! SaveModelStats ( m_activeFileName , m_activeFileCommon ) )
Print ( ID + " : ERROR - online-learning SaveModelStats failed for " + m_activeFileName + " . " ) ;
// Only reset the counter on a successful weight save - resetting unconditionally on a
// transient failure would silently double the effective data-loss window on the NEXT failure too.
if ( saveOk )
{
m_onlineBarsSincePersist = 0 ;
PrintVerbose ( ID + " : online-learning checkpoint saved ( " + IntegerToString ( ( int ) m_onlineSamples )
+ " total updates, rolling acc " + DoubleToString ( m_onlineRollingAcc , 1 ) + " %) " ) ;
}
}
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
void CExpertSignalAIBase : : SaveShadowNet ( const double & indicatorParams [ ] )
{
if ( CheckPointer ( m_shadowNet ) = = POINTER_INVALID )
return ;
m_shadowNet . Save ( m_activeFileName + " _shadow.nnw " , dError , dUndefine , dForecast , dtStudied , m_activeFileCommon , m_eraCount , m_trainingComplete , indicatorParams ) ;
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
void CExpertSignalAIBase : : EnsureShadowNet ( void )
{
if ( CheckPointer ( m_shadowNet ) ! = POINTER_INVALID )
return ;
//--- Pure-MQL5 inference (DLL-free backtest): no era blending happens, so the shadow would just be a
//--- copy of Net - and bootstrapping one via Save/Load would spin a compute backend up on the clone,
//--- defeating the DLL-free goal. Skip it; RefreshLatestSignal() falls back to Net directly.
if ( CheckPointer ( Net ) ! = POINTER_INVALID & & Net . CpuInference ( ) )
return ;
string shadowFile = m_activeFileName + " _shadow.nnw " ;
if ( FileIsExist ( shadowFile , m_activeFileCommon ? FILE_COMMON : 0 ) )
{
CNet * loaded = new CNet ( NULL ) ;
if ( CheckPointer ( loaded ) ! = POINTER_INVALID )
{
double loadE , loadU , loadF ;
datetime loadTime ;
long loadEra ;
bool loadComplete ;
double loadIp [ ] ;
if ( loaded . Load ( shadowFile , loadE , loadU , loadF , loadTime , m_activeFileCommon , loadEra , loadComplete , loadIp , true /*quiet: a miss just falls through to the clone bootstrap below*/ ) )
{
m_shadowNet = loaded ;
//--- This second CNet spins up its OWN compute backend, so on a fresh attach the log shows a
//--- second backend-init block right after the main model's. Name it here (verbose) so it
//--- reads as "the shadow net came up" rather than "the EA started twice".
PrintVerbose ( ID + " : EMA shadow net restored from " + shadowFile + " (its own network instance - hence a second compute-backend init) " ) ;
return ;
}
delete loaded ;
}
}
//--- No compatible persisted shadow - bootstrap from Net's current weights. Clone via the full
//--- Save()/Load() pair - see StartOosContinualSimulation()'s matching comment for why a lighter
//--- restore is not safe here (it would need opencl/directml already initialized on the target
//--- CNet, which a bare "new CNet(NULL)" does not have).
if ( CheckPointer ( Net ) = = POINTER_INVALID )
return ;
//--- Attempt the clone bootstrap at most once per topology (see m_shadowBootstrapAttempted). On the
//--- tester's CPU-DLL fallback a second full-net clone can fail to load; retrying every bar would
//--- rebuild the compute backend each tick and crawl. Falling back to Net is correct and lossless here.
if ( m_shadowBootstrapAttempted )
return ;
m_shadowBootstrapAttempted = true ;
//--- Co-locate the ephemeral clone temp with the active model (COMMON on a live chart, LOCAL in the
//--- tester sandbox) instead of always LOCAL.
string cloneFile = m_activeFileName + " _shadowclone.tmp " ;
int cloneFlags = m_activeFileCommon ? FILE_COMMON : 0 ;
double ip [ ] ;
if ( ! Net . Save ( cloneFile , 0.0 , 0.0 , 0.0 , dtStudied , m_activeFileCommon , m_eraCount , m_trainingComplete , ip ) )
return ;
CNet * clone = new CNet ( NULL ) ;
if ( CheckPointer ( clone ) = = POINTER_INVALID )
{
FileDelete ( cloneFile , cloneFlags ) ;
return ;
}
double loadE , loadU , loadF ;
datetime loadTime ;
long loadEra ;
bool loadComplete ;
double loadIp [ ] ;
bool loaded = clone . Load ( cloneFile , loadE , loadU , loadF , loadTime , m_activeFileCommon , loadEra , loadComplete , loadIp , true /*quiet: best-effort clone, the miss is handled gracefully below*/ ) ;
FileDelete ( cloneFile , cloneFlags ) ;
if ( ! loaded )
{
delete clone ;
//--- Best-effort: without a shadow, live signals read the main Net directly (RefreshLatestSignal's
//--- deployNet fallback), which is correct and lossless - so one calm line, not an error.
//--- This used to be described as EXPECTED on a CPU-DLL box ("can't allocate a 2nd net"). It is not:
//--- the clone load was failing for the same reason the MAIN model load was - CLayer::CreateElement
//--- had stopped overriding CArrayObj::CreateElement, so every CNet::Load failed at layer 0
//--- regardless of backend (see AI\Network.mqh). With that fixed this path should be rare; if it
//--- shows up repeatedly, investigate rather than assume a hardware limit.
PrintVerbose ( ID + " : EMA shadow net could not be bootstrapped - live signals use the main model directly (lossless fallback). " ) ;
return ;
}
m_shadowNet = clone ;
//--- See the matching note on the restore path above: a second CNet means a second compute-backend init
//--- in the log, which is expected, not a duplicated EA.
PrintVerbose ( ID + " : EMA shadow net bootstrapped from the main model's current weights (its own network instance - hence a second compute-backend init) " ) ;
}
# endif // WARRIOR_AIBASE_ONLINELEARNING_MQH
