animatedread/Warrior_EA
Watch
4
6
Fork
You've already forked Warrior_EA
 7
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
Warrior_EA/Expert/AIBase/Training.mqh
AnimateDread bfc1da9de1 fix: the sequence models were reading the window backwards 
BuildFeatureWindow() replaces eight hand-rolled copies of the same loop
and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first,
because MQL5 timeseries indices run backwards and `r + b` with b ascending
walks into the past.

Harmless for PAI and CONV - a dense layer learns a weight per position
either way, a conv learns time-mirrored kernels. Not harmless for the
recurrent stacks:

  - LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t.
  - It writes output[] only when t == steps-1: the visible output IS the
    last hidden state.
  - c_t = f*c_{t-1} + i*g decays toward the start of the sequence.
    lstm_seq_flowcheck.cpp measured block 0's influence on the output at
    1.2e-2 of block T-1's, at the shipped forget bias of 1.0.

So the bar being PREDICTED sat at the far end of the decay and the output
was handed to the OLDEST bar in the window - the exact inverse of what the
window is for. ~80x backwards on LSTM and HYBRID, on all three tiers
(OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never
surfaced as a backend discrepancy.

This does not create edge - the MI diagnostics read at the noise floor
(p=0.4975) with a working positive control. It makes the one hypothesis
those diagnostics explicitly do NOT cover testable: they are marginal and
per-bar, and state they "cannot rule out one that only exists in
combination or across time". The sequence model is the instrument for
across-time structure and it has been crippled, so that hypothesis has
never been honestly tested.

Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and
its features, so a stale .nnw would load cleanly and run a model fitted to
one ordering against the other, silently. Re-keying every config is the
point, not collateral damage. FORCES A FULL RETRAIN.

Also: the now-relative bar caches are re-keyed on the two live paths.
EnsureBarCachesCapacity() was only ever called from training paths, but
once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar
to RefreshConvergedSignal() and Train() is never re-entered - so nothing
cleared the feature cache again for the life of the process. A chart that
trained to convergence kept replaying the rows computed for the last
training era's bar grid: the live signal froze at its convergence-time
value, and OnlineLearnStep() backpropped those stale features against
freshly resolved labels. Backtests were never affected (an inference-only
process never allocates the arrays, so every read recomputes).

Compiles clean: 0 errors, 0 warnings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 18:28:44 -04:00

1794 lines
126 KiB
MQL5
Raw Permalink Blame History

//+------------------------------------------------------------------+
//| Warrior_EA |
//| AnimateDread |
//| |
//| Era loop, plateau ladder, checkpoint selection, deploy/finalise.|
//| |
//| PARTIAL IMPLEMENTATION FILE - not standalone. |
//| This holds CExpertSignalAIBase method BODIES only. The class |
//| declaration lives in Expert\ExpertSignalAIBase.mqh, which |
//| #includes this file at the bottom, after the declaration. Do not |
//| include it anywhere else and do not compile it on its own. |
//| |
//| Split out purely to make the 8216-line original navigable; the |
//| code inside was moved verbatim, not rewritten. |
//+------------------------------------------------------------------+
#ifndef WARRIOR_AIBASE_TRAINING_MQH
#define WARRIOR_AIBASE_TRAINING_MQH
//+------------------------------------------------------------------+
//| Training and Signal Methods |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::Train(datetime StartTrainBar = 0)
{
//--- One-shot latch so a failing forward pass reports itself ONCE per call instead of once per
//--- sample. CNet::feedForward's return value used to be discarded at all three call sites below,
//--- which is how the 2026-08-02 run spent a whole era backpropagating against a batch-norm layer
//--- whose device-side output had frozen: the only trace was 13,776 identical BufferWrite lines
//--- from three frames deeper, and nothing said training was still running on top of them.
bool forwardFailureReported = false;
const int STABILITY_WINDOW = 3; // consecutive eras the OOS accuracy must hold steady for
const double STABILITY_TOLERANCE = 2.0; // max spread (percentage points) across that window
// Max wall-clock work per call before yielding - see m_trainRunActive's declaration comment for
// why chunking exists at all. TuneIndicatorsAndTrain()/Train() only run ONCE per dispatched
// "New Bar" custom chart event (see OnChartEventHandler - id 1001 calls it exactly once, then
// clears bEventStudy so ScheduleTrainingIfNeeded() can arm the next one), so the real throughput
// ceiling in practice is however fast MT5 itself pumps/dispatches that custom event - NOT this
// constant. Raising the OnTimer interval (5s->250ms) had ~zero effect for exactly that reason:
// ticks/chart events were already redispatching far more often than the timer alone would. Since
// per-event dispatch overhead is roughly fixed, doing more compute per event (fewer, larger
// chunks) cuts wall-clock training time roughly in proportion, but MT5 has only this one thread -
// the panel/chart can only respond to input in the gap between chunks, so 500ms made it feel
// unresponsive unless clicks landed in that narrow window. Lowered back to 120ms to keep the UI
// reactive. Raised to 200ms 2026-07-26 (throughput became the bigger complaint, as flagged above) -
// a deliberate middle ground between the reactive-but-slow 120ms and the previously-rejected 500ms,
// not a return to that. Watch panel drag/click feel after this change; back off toward 120ms if it
// regresses, or raise further only in small steps if it doesn't.
// 2026-07-30: it regressed, exactly as that warning anticipated - the panel drags stickily and
// buttons miss clicks under load, because 200ms is the worst-case latency between a click landing
// and this thread being free to notice it. Backing off to the documented 120ms. The throughput this
// costs is a far smaller sacrifice than it was when the note above was written: the derived topology
// cut the network from ~292k weights to ~29k (see ComputeFirstLayerWidth), so an era is a fraction
// of the work it used to be and the fixed per-dispatch overhead the note worried about is now a
// correspondingly smaller share of it. Responsiveness is worth more than the remainder.
const uint TRAIN_TIME_BUDGET_MS = 120;
//---
//--- Never block the calling thread while paused/stopped - just decline this call (or finalize a
//--- run that just got stopped) and let the next scheduled call check again, so Pause/Resume/Stop
//--- and everything else on the control panel stays responsive instead of Sleep()-ing the one
//--- MQL5 thread this chart has.
if(m_trainingPaused && !IsStopped() && !m_trainingStopRequested)
return;
bool stop = IsStopped() || m_trainingStopRequested;
if(stop)
{
if(m_trainRunActive)
FinalizeTrainRun();
if(m_simOosRunActive)
{
delete m_simOosNet;
m_simOosNet = NULL;
m_simOosRunActive = false;
}
return;
}
//--- Evaluation-only continual-learning OOS simulation walk in progress (see StartOosContinualSimulation):
//--- give it exclusive occupancy of this call, same chunked budget as the real era loop below, so a
//--- large OOS window can't freeze the UI in one shot. While it's active no real-training
//--- ResizeBuffers()/RefreshData() runs, so the price/ATR/time buffers it reads stay frozen for its
//--- whole walk - it never has to worry about the label cache's shifting-index invalidation below.
if(m_simOosRunActive)
{
AdvanceOosSimulationChunk();
return;
}
//--- Eager label-cache pre-build in progress (see StartLabelCachePrebuild/AdvanceLabelCachePrebuild) -
//--- same exclusive-occupancy/chunking treatment as the OOS simulation walk above, so it can't freeze
//--- the UI on a large study window either. m_trainRunActive stays false for its whole duration, so
//--- once it completes, Train() falls through to the normal !m_trainRunActive setup below and era 0
//--- starts from the measured class distribution it just seeded.
if(m_labelPrebuildActive)
{
AdvanceLabelCachePrebuild();
return;
}
if(!m_trainRunActive)
{
//--- Wait (briefly, bounded, non-blocking across calls) for the terminal to finish syncing this
//--- symbol/period's history from the broker before computing the training window. Bars(symbol,
//--- period) - the hard cap on how many bars the era loop below will ever process - reflects
//--- whatever's synced SO FAR, not necessarily the true total; starting before sync completes
//--- would let that cap (and therefore the "Bar X of Y" progress display) silently grow between
//--- eras as more history trickles in.
if(!SeriesInfoInteger(m_symbol.Name(), PERIOD_CURRENT, SERIES_SYNCHRONIZED))
{
uint syncNowTick = GetTickCount();
if(m_syncWaitStartTick == 0)
m_syncWaitStartTick = syncNowTick;
if(syncNowTick - m_syncWaitStartTick < 5000)
return; // retry on the next scheduled call instead of blocking here
Print(ID + ": WARNING - history for " + m_symbol.Name() + " " + EnumToString(PERIOD_CURRENT) + " did not finish syncing after 5s; training window may still grow as more history arrives");
}
m_syncWaitStartTick = 0;
//--- 3 no-op passes before the era loop ever runs for a fresh start (see m_warmupPassesRemaining's
//--- declaration comment) - each is its own separately-scheduled Train() call (this whole method
//--- just returns, deferring to the next "New Bar"/timer-driven call), giving MT5's history sync
//--- several real, wall-clock-separated chances to settle on top of the 5s soft wait just above,
//--- before training commits to a bar count and starts populating the label cache below.
if(m_warmupPassesRemaining > 0)
{
m_warmupPassesRemaining--;
PrintVerbose(ID + ": warm-up pass " + IntegerToString(3 - m_warmupPassesRemaining) + " of 3 (letting history sync settle before training starts)");
return;
}
//--- ALL available history, floored by MinTrainYear. The StudyPeriods input this replaces could only
//--- ever throw data away: the signal is weak and the directional classes are rare, so every extra
//--- year is more of the minority class, and the honest generalization read comes from the OOS
//--- holdout rather than from withholding history from training. MinTrainYear survives because it
//--- answers a different question - excluding a broker's dubious pre-history - not "how much".
//--- Note the ordering: SERIES_FIRSTDATE is the floor of what EXISTS, MinTrainYear the floor of what
//--- is TRUSTED, and the window starts at whichever is later.
datetime firstAvailableBar = (datetime)SeriesInfoInteger(m_symbol.Name(), PERIOD_CURRENT, SERIES_FIRSTDATE);
MqlDateTime floor_time;
TimeCurrent(floor_time);
floor_time.year = m_minTrainYear;
floor_time.mon = 1;
floor_time.day = 1;
floor_time.hour = 0;
floor_time.min = 0;
floor_time.sec = 0;
datetime st_time = StructToTime(floor_time);
if(firstAvailableBar > st_time)
st_time = firstAvailableBar;
dtStudied = MathMax(StartTrainBar, st_time);
//--- OOS-based objective + stability tracking: training only "converges" once the objective is
//--- met AND OOS accuracy has held inside a tight band for the last few eras, so a single lucky
//--- era can't get locked in as the final model. The best-scoring era's weights are checkpointed
//--- to an agent-local scratch file (not FILE_COMMON) and restored at the end - this works inside
//--- the tester too, unlike Net.Save()/Load() which are disabled there.
m_oosWindow.Clear();
m_bestOosForecast = -1;
m_bestBalancedOos = -1;
m_bestPassedRecall = false;
m_haveOosCheckpoint = false;
m_oosStable = false;
m_objectiveMet = false;
m_erasSinceCooldown = 0;
m_eraResumePending = false;
//--- Plateau ladder starts fresh with this run, so it re-walks the escalation from its own
//--- starting point. (The focal-gamma anneal that used to reset here went with focal loss on
//--- 2026-07-31 - the ladder's real escape is the learning-rate warm restart.)
m_erasSinceBestBalanced = 0;
m_plateauStage = 0;
//--- One-time eager pre-scan for a fresh start (see m_labelCachePrebuilt's declaration comment) -
//--- kick it off and defer era 0 until it's done, so era 0 can start with a real class-balance
//--- oversampling ratio instead of the reps=1 fallback. Routed via the m_labelPrebuildActive gate
//--- above on every subsequent call until it completes.
if(!m_labelCachePrebuilt)
{
StartLabelCachePrebuild();
return;
}
m_trainRunActive = true;
}
int bars, totalIter, oosCutoff, i;
bool add_loop;
if(!m_eraResumePending)
{
int barsNow = (int)MathMin(Bars(m_symbol.Name(), PERIOD_CURRENT, dtStudied, TimeCurrent()) + m_historyBars, Bars(m_symbol.Name(), PERIOD_CURRENT));
if(!ResizeBuffers(barsNow) || !RefreshData())
{
FinalizeTrainRun();
return;
}
bars = barsNow;
//--- Cross-asset panel is indexed against exactly this bar grid, so it is (re)built wherever
//--- the grid is - never per bar. Non-fatal on failure; see BuildCrossAssetPanel().
BuildCrossAssetPanel(barsNow);
EnsureSpreadSeries(barsNow);
add_loop = false;
//--- Label/feature cache invalidation: MQL5 timeseries indices are always relative to "now"
//--- (index 0 = current bar), so every new closed candle shifts every older bar's index - a
//--- cache keyed by index would silently misalign the moment that happens. See
//--- EnsureBarCachesCapacity() for why `bars` + m_Time.GetData(0) are the correct/sufficient
//--- invalidation keys.
//--- When a wipe happens MID-RUN (a new candle closed while training was still going - e.g. the
//--- market reopening after the weekend), the label cache comes back empty and the lazy per-bar
//--- fallback (ComputeLabelForBar) labels everything Neutral by design (recent pivots are
//--- unconfirmable) - so continuing on a wiped cache silently turns the REST OF THE RUN into
//--- training AND scoring against an all-Neutral world. Observed 2026-07-19: eras 18-20 started
//--- right after the Sunday session open - IS error collapsed 0.44->0.22, OOS "accuracy" soared
//--- to 84.9% with Buy/Sell recall n/a and era time halved, the convergence machinery happily
//--- rewarding all-Neutral predictions on 100%-Neutral relabeled truth. Re-arm the same chunked
//--- eager prebuild that seeded era 0 and defer this era until it completes; its completion
//--- re-seeds the class tallies (m_prebuildSeedPending) so the next era's priors reflect the
//--- freshly relabeled window.
if(EnsureBarCachesCapacity(bars) && m_labelCachePrebuilt)
{
StartLabelCachePrebuild();
return;
}
//--- freeze the just-finished era's true class totals for this new era's priors (see
//--- m_prevEraTrueBuyCount's declaration comment) before resetting the live counters below - EXCEPT
//--- right after StartLabelCachePrebuild()/AdvanceLabelCachePrebuild() seeded them for era 0: the
//--- live m_trueBuyCount/Sell/Neutral tally is still all-zero at that point (nothing trained yet),
//--- so copying it here would silently stomp the real upfront tally back to an empty distribution.
if(m_prebuildSeedPending)
m_prebuildSeedPending = false;
else
{
m_prevEraTrueBuyCount = m_trueBuyCount;
m_prevEraTrueSellCount = m_trueSellCount;
m_prevEraTrueNeutralCount = m_trueNeutralCount;
}
//--- Natural class base rates for the live logit-adjusted decision (see AdjustedSignalFromSoftmax):
//--- derived from the same just-finished-era true class totals the oversampling ratio uses, so live
//--- calibrates to exactly the distribution the model was measured against. Both branches above
//--- leave m_prevEraTrue* holding the freshest real tally (prebuild-seeded on era 0, copied here
//--- otherwise), so updating from them here covers both paths.
//--- 2026-08-01: THE `if(!m_evalMode)` GUARD THAT USED TO WRAP THIS IS GONE, because it silently
//--- disabled the entire imbalance correction for the whole auto-tune search. ApplyLogitAdjustment()
//--- immediately below needs measured priors; without them it clears the offsets and returns. In
//--- eval mode the priors were never measured, so every GA candidate - which is to say every era of
//--- a run with AutoTuneIndicators on, the shipped default - trained under PLAIN cross-entropy.
//--- That was invisible while the labels were near-balanced and became a total Neutral collapse the
//--- moment a 2:6 barrier put the majority class at 52.5%: recall Buy 0% / Sell 0% / Neutral 100%
//--- by era 5 on all four topologies, and the panel stuck on "measuring..." because a model that
//--- never calls a direction never accumulates a directional tally.
//--- The guard's stated fear - a search contaminating the deployed calibration - cannot happen:
//--- these priors are measured from the LABEL distribution, and the tuner only perturbs indicator
//--- periods (MA/RSI/MACD/Ichimoku/AD). The barrier label depends on ATR, SL_Mode and TP_Mode, none
//--- of which the search touches, so every candidate sees byte-identical labels and therefore
//--- identical priors. There is nothing for a candidate to contaminate. What the guard actually
//--- protected against is the .stats WRITE, and that is gated separately (eval candidates never
//--- checkpoint - see m_haveOosCheckpoint - and never persist).
UpdateClassPriors(m_prevEraTrueBuyCount, m_prevEraTrueSellCount, m_prevEraTrueNeutralCount);
//--- Re-install the training-time logit offsets from the priors just measured, so this
//--- era's gradient tracks the distribution the era is scored against. Runs in eval mode
//--- too: a GA candidate must train under the same loss as the real run or its score
//--- means nothing - only the PERSISTED calibration is withheld from eval mode.
ApplyLogitAdjustment();
m_countBuySignals = 0;
m_countSellSignals = 0;
m_countNeutralSignals = 0;
m_trueBuyCount = 0;
m_trueSellCount = 0;
m_trueNeutralCount = 0;
m_oosBuyHits = 0;
m_oosBuyTotal = 0;
m_oosSellHits = 0;
m_oosSellTotal = 0;
m_oosNeutralHits = 0;
m_oosNeutralTotal = 0;
m_oosBuyPredicted = 0;
m_oosBuyPredictedHits = 0;
m_oosSellPredicted = 0;
m_oosSellPredictedHits = 0;
m_oosNeutralPredicted = 0;
m_oosNeutralPredictedHits = 0;
m_oosBuyFired = 0;
m_oosBuyFiredHits = 0;
m_oosSellFired = 0;
m_oosSellFiredHits = 0;
ArrayInitialize(m_oosTierFired, 0);
ArrayInitialize(m_oosTierHits, 0);
m_oosConfidenceSum = 0;
// Nearest-to-present slice of this era's bars is held out as OOS and never backprop'd on;
// the rest (older bars) is the IS/training slice.
totalIter = (int)MathMax(bars - MathMax(m_historyBars, 0), 0);
oosCutoff = (int)(MathMax(0, MathMin(100, m_oosSplitPct)) / 100.0 * totalIter);
i = (int)(bars - MathMax(m_historyBars, 0) - 1);
//--- Fresh era: reset pass 2's shuffled-backprop queue (see m_isTrainQueue's declaration
//--- comment). Preallocated to a parity-shaped ESTIMATE, not a hard worst case: at full parity
//--- all 3 classes replicate to ~the majority count, so the queue lands near 3x totalIter -
//--- 4x covers that plus label drift. The queueing block below grows the arrays on demand if an
//--- era ever exceeds the estimate (it used to silently DROP overflow instead - harmless at the
//--- old totalIter*cap sizing, which could never fill, but real data loss now that the measured
//--- ratio, not a small fixed cap, decides the replica count).
ArrayResize(m_isTrainQueue, totalIter * 4);
ArrayResize(m_isTrainQueueWeightScale, totalIter * 4);
ArrayResize(m_isTrainQueuePrimary, totalIter * 4);
m_isTrainQueueCount = 0;
m_isTrainCursor = 0;
m_isPass2Active = false;
m_isPass2Done = false;
m_isPass3Active = false;
//--- Fresh per-era predicted-signal cache for the end-of-era NMS sweep (see PruneDirectionalClusters).
//--- -2 = "not scored this era" so stale bars from a longer prior era can't draw phantom arrows.
if(m_signalClusterWindow > 0)
{
ArrayResize(m_arrowSignalCache, bars);
ArrayInitialize(m_arrowSignalCache, -2.0);
}
}
else
{
//--- resuming a chunk that yielded mid-bar-loop last call - pick up exactly where it left off
bars = m_resumeBars;
totalIter = m_resumeTotalIter;
oosCutoff = m_resumeOosCutoff;
add_loop = m_resumeAddLoop;
i = m_resumeBarIndex;
m_eraResumePending = false;
}
// Restore this model's own learning-rate trajectory into the shared global right before this
// chunk's backProp() calls touch it - see m_modelEta's declaration comment.
eta = m_modelEta;
uint chunkStartTick = GetTickCount();
// Iterate over the bars - skipped entirely when resuming straight into pass 2, OR when resuming
// into a still-unfinished pass 3 (see m_isPass2Done's declaration comment for why checking
// m_isPass2Active alone isn't enough to detect the latter case): pass 1 already fully completed
// in an earlier call either way.
if(!m_isPass2Active && !m_isPass2Done)
{
for(; i >= 0 && !stop; i--)
{
//--- Build THIS bar's own feature window and feed it forward BEFORE checking/training against
//--- its label - see r's declaration comment below for why the window must end AT bar i, and
//--- why this must run before the label-check block rather than after: the label check needs
//--- this bar's own freshly-computed prediction, not the previous iteration's (see windowOk).
TempData.Clear();
//--- Window ends AT (includes) bar i itself, extending m_historyBars bars into the past - i.e.
//--- "everything known as of this bar's close." Predicting label(i) - "was THIS bar the
//--- reversal" - from a window that stops short of bar i itself would blind the model to the
//--- most recent price action, which is exactly the information a reversal call most depends
//--- on. Must match RefreshLatestSignal()'s window exactly (r=i there too, i=0), since that's
//--- what actually queries the deployed model live - training on a different window than what
//--- gets queried at inference time would teach the wrong task entirely.
//--- BuildFeatureWindow() owns the Clear/Reserve/loop AND the oldest-bar-first ordering that
//--- the LSTM stacks depend on - see its definition comment.
int r = i;
bool windowOk = false;
double displayNeuron0 = 0, displayNeuron1 = 0, displayNeuron2 = 0;
if(r <= bars)
{
windowOk = BuildFeatureWindow(r);
if(windowOk)
add_loop = true;
}
//--- Determine label/queue-eligibility BEFORE running any feedForward this bar - see
//--- wouldQueue's use below for why. Mirrors the label-check condition this block used to
//--- gate on (moved earlier, unchanged).
bool haveLabel = false, buy = false, sell = false, wouldQueue = false;
if(windowOk && i < (int)(bars - MathMax(m_historyBars, 0) - 1) && i > 1 && m_Time.GetData(i) > dtStudied
&& (m_outputNeuronsCount == 1 || m_outputNeuronsCount == 3))
{
//--- The fractal/swing-confirmation/trend-context label at now-relative index i only depends
//--- on price/ATR history, never on model state, so it's identical every era until a new bar
//--- closes and shifts the index frame (see the cache invalidation check above) - cache it
//--- rather than recomputing from scratch every single era. Usually already populated by
//--- AdvanceLabelCachePrebuild() before era 0 ever starts - this is just a lazy fallback for
//--- any index it didn't cover (e.g. bars/window drifted between prebuild and era 0's start).
if(m_labelCacheHasValue[i])
{
buy = m_labelCacheBuy[i];
sell = m_labelCacheSell[i];
}
else
{
ComputeLabelForBar(i, bars, buy, sell);
m_labelCacheBuy[i] = buy;
m_labelCacheSell[i] = sell;
m_labelCacheHasValue[i] = true;
}
haveLabel = true;
bool isOOS = (i < oosCutoff);
// Embargo: a bar's triple-barrier label is decided by the m_barrierHorizonBars bars that
// follow it (see TripleBarrierLabel()). An IS bar within that distance of the OOS boundary
// therefore carries a label that was only knowable using price action from inside the
// held-out OOS window - purge that narrow band from backprop entirely instead of training
// on it as ordinary IS. Lopez de Prado ch. 7 calls this purging, and it is the whole reason
// a naive train/test split leaks on overlapping-horizon financial labels.
// Was m_swingConfirmationBars + LABEL_WINDOW_BARS, which measured the ZigZag repainting
// delay - the correct quantity for the old target and the wrong one for this label.
int embargoBars = MathMax(m_barrierHorizonBars, 1);
bool isEmbargoed = (!isOOS && i < oosCutoff + embargoBars);
wouldQueue = (!isOOS && !isEmbargoed);
}
//--- Only run this bar's feedForward (and the display/count/chart-draw work that depends on
//--- it) when pass 2 ISN'T about to redo it anyway. A queued bar gets a completely fresh
//--- feedForward moments later in pass 2 (see m_isTrainQueue's declaration comment - other
//--- queued bars ahead of it in the shuffled order may already have updated weights, so pass
//--- 2 can't reuse this scan's result even if it wanted to) - running it here too was one
//--- full forward pass per training sample thrown straight in the trash every era, on top of
//--- the one pass 2 actually needs. The book's SGD (references\neuronetworksbook.pdf, section
//--- 1.4) is one forward+backward pass per training sample, not two.
//--- Display/IS-scoring only on this path, but the same rule applies: getResults() after a
//--- failed pass returns the previous bar's activations, which would be shown on the panel and
//--- counted as this bar's prediction.
if(windowOk && !wouldQueue && Net.feedForward(TempData))
{
Net.getResults(TempData);
if(m_outputNeuronsCount == 1)
dPrevSignal = TempData[0];
else
if(m_outputNeuronsCount == 3)
dPrevSignal = ApplyClassificationSoftmax();
//--- Snapshot the just-computed neuron output(s) for the status label display below, before
//--- the label-check block clears/refills TempData with the target label (Step A always
//--- runs after this point now) - reading TempData directly for display after that would
//--- show the TRUE LABEL of the bar just trained on, not the network's own prediction.
if(TempData.Total() > 0)
displayNeuron0 = TempData[0];
if(TempData.Total() > 1)
displayNeuron1 = TempData[1];
if(TempData.Total() > 2)
displayNeuron2 = TempData[2];
switch(DoubleToSignal(dPrevSignal))
{
case Buy:
m_countBuySignals++;
break;
case Sell:
m_countSellSignals++;
break;
default:
m_countNeutralSignals++;
break;
}
m_lastBarTime = m_Time.GetData(i);
if(i > 0)
{
// NMS on: record only - the era-end sweep is the SOLE renderer, so no raw (un-
// declustered) arrow is ever drawn mid-era. NMS off: draw inline as before.
if(m_signalClusterWindow > 0)
{
if(i < ArraySize(m_arrowSignalCache))
m_arrowSignalCache[i] = dPrevSignal;
}
else
if(DoubleToSignal(dPrevSignal) == Neutral)
DeleteObject(m_lastBarTime);
else
DrawObject(m_lastBarTime, dPrevSignal, m_High.GetData(i), m_Low.GetData(i));
}
UpdateTrainingStatusLabel(
StringFormat("Bar %d of %d -> %.2f%% (scan)", bars - i + 1, bars, (double)(bars - i + 1.0) / bars * 100),
displayNeuron0, displayNeuron1, displayNeuron2, dPrevSignal);
}
if(haveLabel)
{
// True label as an ENUM_SIGNAL, derived directly from the buy/sell bools - not read
// back from TempData, which no longer holds a target at this point at all (see above).
ENUM_SIGNAL trueSignal = buy ? Buy : (sell ? Sell : Neutral);
// Track the true class distribution this era (used below to weight IS oversampling,
// and surfaced in the status label text alongside the predicted-class counts)
switch(trueSignal)
{
case Buy:
m_trueBuyCount++;
break;
case Sell:
m_trueSellCount++;
break;
default:
m_trueNeutralCount++;
break;
}
// OOS scoring used to happen right here, against whatever weights this bar's earlier
// feedForward (this pass) happened to be using - which for era 0 is the network's
// still-untrained cold-start state (100% Neutral - see the output-layer bias seed's
// declaration comment), and for every later era is last era's END-of-training state,
// never THIS era's. That silently gave every era's OOS score a full one-era lag behind
// its own training, and made era 0's OOS score meaningless by construction. OOS scoring
// now happens in its own pass (see m_isPass3Active's declaration comment), AFTER pass 2
// has actually trained on this era's IS data, against a fresh feedForward on each OOS
// bar rather than this scan's now-stale one.
if(wouldQueue)
{
// Queue this bar for pass 2's shuffled backProp instead of training on it here,
// immediately, in strict chronological order - see m_isTrainQueue's declaration
// comment for the full rationale. The predicted-signal counts, the chart-marker draw,
// and the dForecast/dUndefine IS-accuracy update are all computed in pass 2 instead,
// against that bar's own freshly-recomputed confidence - see the matching block right
// after pass 2's Net.feedForward() call.
//
// EVERY BAR IS QUEUED EXACTLY ONCE. Class imbalance is corrected analytically inside
// the gradient by the logit-adjusted loss, not by duplicating minority bars here.
//
// The history is worth keeping, because it is why the data-level approach was
// abandoned rather than merely re-tuned. Four successive versions of oversampling all
// collapsed, in both directions:
// v1 uncapped replication x an independent loss weight (up to ~4.5x total) ->
// Buy-only collapse, OOS ~10%, IS error 0.37->0.57 in 4 eras.
// v2 capped the ratio before splitting it between the two -> mathematically the
// same total correction as pure loss weighting, which had already failed.
// v3 replication alone, capped at 3x against a ~5.3x imbalance -> Neutral collapse,
// Buy/Sell recall 0% for 6 straight eras (2026-07-18).
// v4 replication to ~90% parity (up to 28x) -> measured across six topologies on
// 2026-07-29, every model drove ONE direction to ~50% recall and abandoned the
// other, and which direction was arbitrary. One era in 1,301 cleared the floor.
// The through-line: replication makes Buy and Sell compete for the same replicated
// capacity, and Adam's mt/sqrt(vt) normalisation (Kingma & Ba 2015) is near-invariant
// to the gradient rescaling that the loss-weighted variants relied on. Per Buda, Maki
// & Mazurowski 2018, stacking data-level and cost-level corrections on one axis is not
// reliably additive - and the logit-adjusted loss replaces BOTH with a single
// correction that is provably consistent for balanced error.
//
// Pass 2 still Fisher-Yates shuffles the queue: chronological order correlates
// consecutive gradients, which is the same correlated-momentum overshoot documented at
// AI\Network.mqh's MAX_WEIGHT_DELTA comment. That reason is independent of replication
// and survives it.
//--- MINORITY REPLAY REMOVED 2026-07-31. Every bar is queued exactly once; class
//--- imbalance is corrected analytically in the gradient by the logit-adjusted loss
//--- (Menon et al. 2021) instead of by duplicating rare bars in the data. Stacking the
//--- two double-counts the same imbalance - Buda et al. 2018 - and the replay branch had
//--- in fact been gated OFF for the whole shipped configuration, so this is the code
//--- catching up with the behaviour rather than a change in it. Measured 2026-07-29
//--- across six topologies, replay made Buy and Sell compete for the same replicated
//--- capacity: every model drove ONE direction to ~50% recall and abandoned the other,
//--- and which direction was arbitrary. One era in 1,301 cleared the per-class floor.
int repCount = 1;
double perOccurrenceScale = 1.0;
//--- grow on demand (reserve keeps this amortized-rare) - the prealloc above is an
//--- estimate, and dropping overflow would silently starve exactly the minority
//--- classes the replication exists to protect
if(m_isTrainQueueCount + repCount > ArraySize(m_isTrainQueue))
{
int newQueueSize = m_isTrainQueueCount + repCount;
ArrayResize(m_isTrainQueue, newQueueSize, 16384);
ArrayResize(m_isTrainQueueWeightScale, newQueueSize, 16384);
ArrayResize(m_isTrainQueuePrimary, newQueueSize, 16384);
}
for(int rep = 0; rep < repCount; rep++)
{
m_isTrainQueue[m_isTrainQueueCount] = i;
m_isTrainQueueWeightScale[m_isTrainQueueCount] = perOccurrenceScale;
//--- rep 0 is this bar's single "counts once" occurrence - see m_isTrainQueuePrimary.
//--- Every rep still trains; only the reported IS accuracy looks at this flag.
m_isTrainQueuePrimary[m_isTrainQueueCount] = (rep == 0);
m_isTrainQueueCount++;
}
}
}
stop = IsStopped() || m_trainingStopRequested;
if(!stop && i > 0 && GetTickCount() - chunkStartTick >= TRAIN_TIME_BUDGET_MS)
{
//--- yield: save exactly enough to resume this same era, mid-bar-loop, on the next call -
//--- see m_trainRunActive's declaration comment for why this must happen instead of
//--- letting one era (or the whole run) process synchronously to completion
m_resumeBars = bars;
m_resumeTotalIter = totalIter;
m_resumeOosCutoff = oosCutoff;
m_resumeAddLoop = add_loop;
m_resumeBarIndex = i - 1;
m_eraResumePending = true;
// Save this model's own learning-rate trajectory back out of the shared global before
// yielding - see m_modelEta's declaration comment.
m_modelEta = eta;
return;
}
}
} // end if(!m_isPass2Active) - pass 1
//--- Pass 2: replay the bars pass 1 queued into m_isTrainQueue for backProp, in a freshly
//--- shuffled order - see m_isTrainQueue's declaration comment for the full rationale. Runs
//--- whenever pass 1 just finished (or we resumed straight into an already-active pass 2 - see
//--- m_isPass2Active's declaration comment); skipped on a stopped run, an era with no valid window
//--- at all (add_loop still false), or - critically - a resume into a still-unfinished pass 3 (see
//--- m_isPass2Done's declaration comment): without this last check, that resume would re-shuffle
//--- and replay the ENTIRE queue again from scratch every single call.
if(!stop && add_loop && !m_isPass2Done)
{
if(!m_isPass2Active)
{
m_isPass2Active = true;
m_isTrainCursor = 0;
// 2026-07-28: a "replay-only optimizer override" was removed from here. It captured every
// neuron's optimizer and forced the whole net to SGD for the duration of pass 2, on the
// rationale that oversampled minority bars should not "exploit the same Adam-style momentum
// path as the base training pass". But pass 2 IS the base training pass - it is the only place
// Net.backProp() is called during training at all (pass 1 only feeds forward and queues) - so
// the override applied to 100% of weight updates, not to some replay subset.
// Adam's mt/vt were therefore never updated and its bias-correction step counter never
// advanced: TrainingOptimizer=ADAM was silently a no-op and the model trained purely on
// SGD+momentum at Adam's learning rate. It arrived with the DFA change set and was never part
// of any validated run. The optimizer the user selects is now the optimizer that runs.
// Fisher-Yates shuffle - a fresh random order every era, so Adam's momentum can't keep
// landing on the same contiguous same-class label run at the same point in the sequence every
// single era. Barrier labels make those runs LONGER than the old exact-pivot ones (adjacent
// bars share most of their forward window, so they usually resolve the same way), which makes
// the shuffle matter more here, not less. m_isTrainQueueWeightScale is swapped in lockstep - each
// slot's stored per-occurrence weight (see the queueing block's oversampling comment) must
// stay attached to the same bar index it was computed for.
for(int sIdx = m_isTrainQueueCount - 1; sIdx > 0; sIdx--)
{
int sJ = MathRand() % (sIdx + 1);
int sTmp = m_isTrainQueue[sIdx];
m_isTrainQueue[sIdx] = m_isTrainQueue[sJ];
m_isTrainQueue[sJ] = sTmp;
double sScaleTmp = m_isTrainQueueWeightScale[sIdx];
m_isTrainQueueWeightScale[sIdx] = m_isTrainQueueWeightScale[sJ];
m_isTrainQueueWeightScale[sJ] = sScaleTmp;
//--- the primary flag must travel with its own slot too, or the "count this bar once"
//--- marker would end up attached to a different bar's occurrence - see m_isTrainQueuePrimary
bool sPrimTmp = m_isTrainQueuePrimary[sIdx];
m_isTrainQueuePrimary[sIdx] = m_isTrainQueuePrimary[sJ];
m_isTrainQueuePrimary[sJ] = sPrimTmp;
}
}
for(; m_isTrainCursor < m_isTrainQueueCount; m_isTrainCursor++)
{
int qi = m_isTrainQueue[m_isTrainCursor];
bool qWindowOk = BuildFeatureWindow(qi);
//--- A failed forward pass must NOT be followed by backProp() further down this block: the
//--- output layer would still hold the PREVIOUS sample's activations, so the update would be
//--- this bar's label against another bar's prediction - training on pure noise while every
//--- accuracy counter kept reporting normally.
bool qForwardOk = (qWindowOk && TempData.Total() >= (int)m_historyBars * m_neuronsCount &&
Net.feedForward(TempData));
if(qWindowOk && !qForwardOk && !forwardFailureReported)
{
forwardFailureReported = true;
Print(__FUNCTION__ + ": CNet::feedForward FAILED at era " + IntegerToString((int)m_eraCount) +
" - this era's remaining samples are being skipped, not trained. A layer is refusing to"
" accept its own output (check the preceding BufferWrite/BufferRead lines for which"
" buffer, and see NormalizeHost in AI\\NeuronBatchNorm.mqh for the batch-norm case).");
}
if(qForwardOk)
{
Net.getResults(TempData);
// Must go through ApplyClassificationSoftmax() (3-output case) before reading the
// per-class values below - Net.getResults() returns each output neuron's own independent
// SIGMOID activation (each already in [0,1] but NOT summing to 1 across the three), not a
// true class-conditional probability distribution; ApplyClassificationSoftmax() is what
// turns that into one (and is also what pass 1/3's displayNeuron0/1/2 already go through).
double qPrevSignal = (m_outputNeuronsCount == 3) ? ApplyClassificationSoftmax() : TempData[0];
double pt0 = (TempData.Total() > 0) ? TempData[0] : 0.0;
double pt1 = (TempData.Total() > 1) ? TempData[1] : 0.0;
double pt2 = (TempData.Total() > 2) ? TempData[2] : 0.0;
bool qBuy = m_labelCacheHasValue[qi] ? m_labelCacheBuy[qi] : false;
bool qSell = m_labelCacheHasValue[qi] ? m_labelCacheSell[qi] : false;
ENUM_SIGNAL qTrueSignal = qBuy ? Buy : (qSell ? Sell : Neutral);
UpdateTrainingStatusLabel(
StringFormat("Training bar %d of %d -> %.2f%% (shuffled)", m_isTrainCursor + 1, m_isTrainQueueCount, (double)(m_isTrainCursor + 1.0) / MathMax(m_isTrainQueueCount, 1) * 100),
pt0, pt1, pt2, qPrevSignal);
//--- Predicted-signal tally, chart marker, and IS-accuracy stat that pass 1 used to compute
//--- from its own (now-removed) redundant feedForward on this same bar - see pass 1's
//--- wouldQueue comment. Uses THIS feedForward's result (the only one this bar gets), so
//--- these now reflect the model's state as of this bar's own turn in the shuffled replay
//--- (post any earlier-shuffled bar's backProp this era), not a separate pre-training
//--- snapshot - matching how a standard shuffled-epoch SGD run reports running training
//--- accuracy during the epoch rather than in a discarded pre-epoch dry run.
switch(DoubleToSignal(qPrevSignal))
{
case Buy:
m_countBuySignals++;
break;
case Sell:
m_countSellSignals++;
break;
default:
m_countNeutralSignals++;
break;
}
datetime qBarTime = m_Time.GetData(qi);
// NMS on: record only (the era-end sweep renders); off: draw inline. See pass 1's note.
if(m_signalClusterWindow > 0)
{
if(qi < ArraySize(m_arrowSignalCache))
m_arrowSignalCache[qi] = qPrevSignal;
}
else
if(DoubleToSignal(qPrevSignal) == Neutral)
DeleteObject(qBarTime);
else
DrawObject(qBarTime, qPrevSignal, m_High.GetData(qi), m_Low.GetData(qi));
bool qClassified = (DoubleToSignal(qPrevSignal) == Buy || DoubleToSignal(qPrevSignal) == Sell || DoubleToSignal(qPrevSignal) == Neutral);
if(qClassified)
{
bool isHit = (DoubleToSignal(qPrevSignal) == qTrueSignal);
if(isHit)
dForecast += (100 - dForecast) / Net.recentAverageSmoothingFactor;
else
dForecast -= dForecast / Net.recentAverageSmoothingFactor;
dUndefine -= dUndefine / Net.recentAverageSmoothingFactor;
//--- Compounded, persistent DIRECTIONAL win-rate: count only bars the model actually called
//--- Buy or Sell (a Neutral "no trade" call is neither a win nor a loss), so this tracks the
//--- accuracy of its directional signals rather than the Neutral-inflated all-class rate.
//--- ...and count each BAR once, not each oversampled OCCURRENCE (m_isTrainQueuePrimary):
//--- the queue duplicates minority bars up to ~21x, so counting every occurrence scored this
//--- metric over a ~58%-directional set while its OOS counterpart scored the real ~6%
//--- distribution - two numbers that look comparable, aren't, and made a healthy run read as
//--- severe overfitting. See m_isTrainQueuePrimary for the worked example.
ENUM_SIGNAL qPred = DoubleToSignal(qPrevSignal);
if(m_isTrainQueuePrimary[m_isTrainCursor] && (qPred == Buy || qPred == Sell))
{
m_cumIsTotal++;
if(isHit)
m_cumIsCorrect++;
}
}
else
if(qBuy && qSell)
dUndefine += (100 - dUndefine) / Net.recentAverageSmoothingFactor;
TempData.Clear();
if(m_outputNeuronsCount == 1)
TempData.Add(qBuy && !qSell ? 1 : !qBuy && qSell ? -1 : 0);
else
if(m_outputNeuronsCount == 3)
{
TempData.Add(qBuy ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW);
TempData.Add(qSell ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW);
TempData.Add((!qBuy && !qSell) ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW);
}
// Per-slot weight from m_isTrainQueueWeightScale[m_isTrainCursor], decided once at queue
// time in pass 1. Currently always 1.0: class imbalance is corrected analytically inside
// the gradient by the logit-adjusted loss, so there is no per-sample reweighting left to
// apply here at all. Kept as a real per-slot value rather than a literal 1.0 inline so a
// future supplemental weight can be reintroduced without re-touching the queueing or
// shuffle code.
//--- FOCAL-LOSS MODULATION REMOVED 2026-07-31. It multiplied this weight by (1-pt)^gamma,
//--- a second correction on the same axis as the logit adjustment - the stacking failure
//--- Buda et al. 2018 describes and this file already cited in two other places. It was
//--- running at an eighth strength (gamma * 0.125), damped by the replay toggle, for a
//--- replay path that the adjusted loss had already switched off - so the damping was
//--- calibrated against a mechanism that was not running. See the class-imbalance audit in
//--- Variables\Inputs.mqh.
double qSampleWeight = m_isTrainQueueWeightScale[m_isTrainCursor];
Net.backProp(TempData, qSampleWeight);
}
if(m_isTrainCursor + 1 < m_isTrainQueueCount && GetTickCount() - chunkStartTick >= TRAIN_TIME_BUDGET_MS)
{
//--- yield: save enough to resume PASS 2 mid-queue on the next call - m_isPass2Active
//--- and m_isTrainCursor (both members) carry the actual resume position; bars/oosCutoff/
//--- add_loop are stashed the same way pass 1 already does, since era-end logic just
//--- below still needs them once pass 2 finishes.
m_resumeBars = bars;
m_resumeTotalIter = totalIter;
m_resumeOosCutoff = oosCutoff;
m_resumeAddLoop = add_loop;
m_resumeBarIndex = i;
m_eraResumePending = true;
m_modelEta = eta;
return;
}
}
m_isPass2Active = false;
m_isPass2Done = true;
}
//--- Pass 3: OOS scoring, chronological, AFTER pass 2 has actually trained on this era's IS data -
//--- see m_isPass3Active's declaration comment for why this can no longer happen inline during
//--- pass 1's scan.
if(!stop && add_loop)
{
if(!m_isPass3Active)
{
m_isPass3Active = true;
m_oosScoreStartIndex = (int)MathMin(oosCutoff - 1, bars - MathMax(m_historyBars, 0) - 2);
m_oosScoreIndex = m_oosScoreStartIndex;
for(int rn = 0; rn < 3; rn++)
{
m_oosOutMin[rn] = DBL_MAX;
m_oosOutMax[rn] = -DBL_MAX;
}
m_oosOutSpreadSum = 0.0;
m_oosOutCount = 0;
}
for(; m_oosScoreIndex >= 2; m_oosScoreIndex--)
{
int oi = m_oosScoreIndex;
if(!(oi < (int)(bars - MathMax(m_historyBars, 0) - 1) && m_Time.GetData(oi) > dtStudied))
continue;
bool oWindowOk = BuildFeatureWindow(oi);
//--- Same guard as pass 2, and it matters more here: OOS accuracy is what checkpoint selection
//--- and the plateau ladder's auto-deploy both rank on, so scoring a stale forward pass would
//--- not just be wrong, it would be wrong in the one number that decides which model ships.
//--- A skipped bar simply isn't counted; it never becomes a hit or a miss.
bool oForwardOk = (oWindowOk && TempData.Total() >= (int)m_historyBars * m_neuronsCount &&
Net.feedForward(TempData));
if(oWindowOk && !oForwardOk && !forwardFailureReported)
{
forwardFailureReported = true;
Print(__FUNCTION__ + ": CNet::feedForward FAILED during OOS scoring at era " +
IntegerToString((int)m_eraCount) + " - affected bars are excluded from the OOS"
" accuracy rather than scored against a stale prediction.");
}
if(oForwardOk)
{
Net.getResults(TempData);
// Raw output stats MUST be captured here, before ApplyClassificationSoftmax() overwrites
// TempData[0..2] in place with the softmax probabilities - see m_oosOutMin's declaration
// comment for what these feed.
if(m_outputNeuronsCount == 3 && TempData.Total() >= 3)
{
double rawHi = -DBL_MAX, rawLo = DBL_MAX;
for(int rn = 0; rn < 3; rn++)
{
double rv = TempData.At(rn);
if(rv < m_oosOutMin[rn])
m_oosOutMin[rn] = rv;
if(rv > m_oosOutMax[rn])
m_oosOutMax[rn] = rv;
rawHi = MathMax(rawHi, rv);
rawLo = MathMin(rawLo, rv);
}
m_oosOutSpreadSum += rawHi - rawLo;
m_oosOutCount++;
}
double oPrevSignal = (m_outputNeuronsCount == 3) ? ApplyClassificationSoftmax() : TempData[0];
double oDeploySignal = oPrevSignal;
if(m_outputNeuronsCount == 3)
oDeploySignal = AdjustedSignalFromSoftmax();
double oNeuron0 = (TempData.Total() > 0) ? TempData[0] : 0.0;
double oNeuron1 = (TempData.Total() > 1) ? TempData[1] : 0.0;
double oNeuron2 = (TempData.Total() > 2) ? TempData[2] : 0.0;
bool oBuy = m_labelCacheHasValue[oi] ? m_labelCacheBuy[oi] : false;
bool oSell = m_labelCacheHasValue[oi] ? m_labelCacheSell[oi] : false;
ENUM_SIGNAL oTrueSignal = oBuy ? Buy : (oSell ? Sell : Neutral);
UpdateTrainingStatusLabel(
StringFormat("Scoring OOS bar %d of %d -> %.2f%% (post-training)", m_oosScoreStartIndex - m_oosScoreIndex + 1, m_oosScoreStartIndex + 1,
(double)(m_oosScoreStartIndex - m_oosScoreIndex + 1.0) / MathMax(m_oosScoreStartIndex + 1, 1) * 100),
oNeuron0, oNeuron1, oNeuron2, oDeploySignal);
// Held-out bar: score the model's freshly-trained-this-era forecast against the actual
// outcome without learning from it - keeps the OOS accuracy an honest overfitting signal.
bool oClassified = (DoubleToSignal(oPrevSignal) == Buy || DoubleToSignal(oPrevSignal) == Sell || DoubleToSignal(oPrevSignal) == Neutral);
if(oClassified)
{
m_oosSamples++;
m_oosConfidenceSum += MathAbs(oPrevSignal);
if(dOosError < 0)
dOosError = 0;
bool hit = (DoubleToSignal(oPrevSignal) == oTrueSignal);
//--- Compounded, persistent DIRECTIONAL win-rate: count only bars the model actually called
//--- Buy or Sell (Neutral "no trade" calls aren't wins or losses), so this reflects the
//--- accuracy of its directional signals, not the Neutral-inflated all-class rate.
ENUM_SIGNAL oPred = DoubleToSignal(oPrevSignal);
if(oPred == Buy || oPred == Sell)
{
m_cumOosTotal++;
if(hit)
m_cumOosCorrect++;
}
// Per-class confusion counts, used for the Buy/Sell recall convergence gate below
switch(oTrueSignal)
{
case Buy:
m_oosBuyTotal++;
if(hit)
m_oosBuyHits++;
break;
case Sell:
m_oosSellTotal++;
if(hit)
m_oosSellHits++;
break;
default:
m_oosNeutralTotal++;
if(hit)
m_oosNeutralHits++;
break;
}
// Same confusion counts keyed by what the model actually PREDICTED this bar, not the
// true label - see m_oosBuyPredicted's declaration comment for why recall alone can
// hide an over-firing class.
switch(DoubleToSignal(oPrevSignal))
{
case Buy:
m_oosBuyPredicted++;
if(hit)
m_oosBuyPredictedHits++;
break;
case Sell:
m_oosSellPredicted++;
if(hit)
m_oosSellPredictedHits++;
break;
default:
m_oosNeutralPredicted++;
if(hit)
m_oosNeutralPredictedHits++;
break;
}
// Live-decision precision: scores the bars on which the deployed EA would actually cast a
// directional vote, using the prior-corrected (logit-adjusted) posterior - see
// AdjustedSignalFromSoftmax()/RefreshLatestSignal(). The recall/argmax-precision above stay
// on the raw argmax (the model's intrinsic class separation, which the convergence gate
// needs); THIS scores what trades live, so the panel's live precision number is the
// precision a buyer gets forward. TempData still holds this bar's raw softmax probs
// (nothing overwrote them since ApplyClassificationSoftmax above), so the adjustment reads
// them directly. Neutral picks aren't counted - they cast no vote.
// No confidence-floor term any more: with the floor removed, EVERY non-Neutral adjusted
// decision casts a vote (at its tier weight), so any threshold here would score a
// different population than the one that actually votes. Whether a given vote goes on to
// OPEN a position additionally depends on Min_Vote_Open versus the AVERAGE across all
// voting filters, which this per-bar training-time scorer has no visibility of - so this
// stays the honest "would have voted, and was it right" measure rather than pretending to
// model the aggregate.
if(m_outputNeuronsCount == 3)
{
double adjSig = AdjustedSignalFromSoftmax();
ENUM_SIGNAL adjEnum = DoubleToSignal(adjSig);
if(adjEnum != Neutral)
{
bool fireHit = (adjEnum == oTrueSignal);
//--- Bucket the same fire by confidence tier - see m_oosTierFired. Safe to call here
//--- and nowhere earlier: ConfidenceTier() reads the net's CURRENT outputs, which is
//--- exactly the bar AdjustedSignalFromSoftmax() just scored.
int fireTier = ConfidenceTier();
if(fireTier >= 0 && fireTier < 4)
{
m_oosTierFired[fireTier]++;
if(fireHit)
m_oosTierHits[fireTier]++;
}
if(adjEnum == Buy)
{
m_oosBuyFired++;
if(fireHit)
m_oosBuyFiredHits++;
}
else
{
m_oosSellFired++;
if(fireHit)
m_oosSellFiredHits++;
}
}
}
if(hit)
{
dOosForecast += (100 - dOosForecast) / Net.recentAverageSmoothingFactor;
dOosError -= dOosError / Net.recentAverageSmoothingFactor;
}
else
{
dOosForecast -= dOosForecast / Net.recentAverageSmoothingFactor;
dOosError += (100 - dOosError) / Net.recentAverageSmoothingFactor;
}
}
// Mirror pass 1's chart annotation for this (OOS) bar, now using post-training weights
// instead of pass 1's pre-training snapshot - last write wins on the shared per-bar-time
// object, so this supersedes pass 1's earlier draw for the same bar with the correct one.
m_lastBarTime = m_Time.GetData(oi);
if(oi > 0)
{
// NMS on: record only (the era-end sweep renders); off: draw inline. See pass 1's note.
if(m_signalClusterWindow > 0)
{
if(oi < ArraySize(m_arrowSignalCache))
m_arrowSignalCache[oi] = oDeploySignal;
}
else
if(DoubleToSignal(oDeploySignal) == Neutral)
DeleteObject(m_lastBarTime);
else
DrawObject(m_lastBarTime, oDeploySignal, m_High.GetData(oi), m_Low.GetData(oi));
}
}
if(m_oosScoreIndex - 1 >= 2 && GetTickCount() - chunkStartTick >= TRAIN_TIME_BUDGET_MS)
{
//--- yield: save enough to resume PASS 3 mid-walk on the next call - m_isPass3Active and
//--- m_oosScoreIndex (both members) carry the actual resume position.
m_resumeBars = bars;
m_resumeTotalIter = totalIter;
m_resumeOosCutoff = oosCutoff;
m_resumeAddLoop = add_loop;
m_resumeBarIndex = i;
m_eraResumePending = true;
m_modelEta = eta;
return;
}
}
m_isPass3Active = false;
//--- Pass 3 done => every scored bar's prediction is now in m_arrowSignalCache. Collapse each
//--- same-direction cluster to its earliest bar so the chart shows one arrow per real turn.
PruneDirectionalClusters(bars);
}
//--- Diagnostic recall snapshot for the periodic progress log further below - populated inside
//--- the m_oosSamples>0 recall-gate block when this era actually computes it; stays -1 ("n/a"
//--- in the log) on eras that don't (era 0, or a stopped/cap-hit era).
int logBuyRecallPct = -1, logSellRecallPct = -1, logNeutralRecallPct = -1;
//--- Balanced accuracy (macro-recall) this era, surfaced in the log so the metric the checkpoint
//--- is now selected on is visible - see m_bestBalancedOos. -1 ("n/a") on eras that don't score.
int logBalancedAccPct = -1;
int logCoveragePct = -1;
int logDirPrecPct = -1;
//--- Zero-skill precision for this era's label mix - see chancePrecPct. Logged beside the selection
//--- score because the raw precision number is meaningless without it: 44% is excellent against a
//--- 3% chance level and worthless against a 43% one, and the whole 2026-08-01 confusion was
//--- reading the first as if it were the second.
int logChancePrecPct = -1;
//--- Predicted-rate (of all OOS bars this era, how often the model called this class at all) and
//--- precision (of the calls it made, how many were right) for Buy/Sell - m_oosBuyPredicted/
//--- m_oosSellPredicted (see that member's declaration comment) were already being tracked for
//--- exactly this but never surfaced anywhere. A recall-only view can't tell "the model never once
//--- calls Sell" (predicted rate stuck at 0%) apart from "the model calls Sell plenty but always on
//--- the wrong bars" (predicted rate healthy, precision near 0%) - both show up identically as 0%
//--- Sell recall, but point at completely different problems (a suppressed/dead output vs. a
//--- miscalibrated decision boundary), so this splits them out.
int logBuyPredPct = -1, logSellPredPct = -1, logBuyPrecPct = -1, logSellPrecPct = -1;
//--- Live-fired precision (%) per direction this era - the precision on just the bars that cleared
//--- the confidence floor under the live/prior-corrected decision, i.e. what would actually trade.
int logBuyFiredPrecPct = -1, logSellFiredPrecPct = -1;
bool shouldLogProgress = false;
//--- era complete (ran out of bars) or a stop was requested mid-era
if(add_loop)
{
m_eraCount++;
m_erasSinceCooldown++;
//--- EMA shadow-weight deployment: blend the shadow a small step (SHADOW_WEIGHT_TAU) toward
//--- Net's just-updated weights, every era - see m_shadowNet's declaration comment. Must run
//--- here, inside the era loop, not just once at Train()-end: the whole point is damping the
//--- WITHIN-run oscillation (era-to-era whipsaw), which a single end-of-run blend would miss
//--- entirely.
EnsureShadowNet();
if(CheckPointer(m_shadowNet) != POINTER_INVALID)
m_shadowNet.BlendWeightsFrom(Net, SHADOW_WEIGHT_TAU);
//--- Status-label progress is invisible with no chart (headless/optimization runs), and even in
//--- visual mode a long training run can otherwise look "stuck" for a long time with no
//--- Journal output at all - log progress at most every ~5s (real wall-clock, not simulated
//--- time) so an operator can tell it's actively working, not hung. The actual Print() is
//--- deferred past the recall-gate block below (see logBuyRecallPct etc.) so this line can
//--- show per-class OOS recall - once OOS accuracy alone clears the target, recall is the
//--- most common thing still silently blocking convergence, and previously had no visibility
//--- outside of a regression event.
static uint lastProgressLogTick = 0;
uint nowTick = GetTickCount();
shouldLogProgress = (nowTick - lastProgressLogTick >= 5000);
if(shouldLogProgress)
lastProgressLogTick = nowTick;
//--- Era cap. There used to be a second, much smaller cap here for throwaway auto-tune
//--- candidates; the filter tuner does not train candidates at all, so only the real one remains.
int effectiveEraCap = m_maxErasPerRun;
//--- PLATEAU LADDER, terminal stage: training stopped improving and both escape attempts (two
//--- learning-rate warm restarts) failed to find anything better - see the ladder in the era-end block
//--- below, which is what raised m_plateauStage this far and already logged why. This is the
//--- normal, expected way a run finishes now that there is no absolute accuracy target to hit:
//--- it trains until it genuinely stops getting better, then deploys its best checkpoint.
//--- Same mechanism as the operator's "No" answer at the era cap (see that branch's comments for
//--- why m_trainingComplete is set here and why m_trainingStopRequested deliberately is NOT):
//--- stop ends this era loop, FinalizeTrainRun() then restores and deploys the best checkpoint.
if(m_plateauStage >= PLATEAU_STAGE_DEPLOY && m_bestPassedRecall && m_haveOosCheckpoint)
{
stop = true;
m_trainingComplete = true;
}
else if(effectiveEraCap > 0 && m_erasSinceCooldown >= effectiveEraCap)
{
//--- Era cap reached without converging: ask the operator whether to keep training or
//--- deploy the best checkpoint and stop (see PromptContinuePastEraCap / m_maxErasPerRun).
if(PromptContinuePastEraCap(dOosForecast))
{
m_erasSinceCooldown = 0; // keep training - reset the cap window
Print(ID + ": hit the " + IntegerToString(m_maxErasPerRun) + "-era cap (best dir-precision " + DoubleToString(m_bestBalancedOos, 1) + "%, blended OOS " + DoubleToString(dOosForecast, 1) + "%) - CONTINUING training by operator choice.");
}
else
{
//--- stop: end THIS era loop now; FinalizeTrainRun (reached via the stop path below,
//--- because stop==true) deploys the best checkpoint. Deliberately do NOT set
//--- m_trainingStopRequested here: m_trainingComplete alone already routes every later
//--- tick to RefreshConvergedSignal (see ScheduleTrainingIfNeeded's if-branch precedence),
//--- so training never re-arms - and leaving m_trainingStopRequested false lets the deployed
//--- model run live inference AND online continual learning IN-SESSION, exactly like a
//--- normal-convergence deploy (which never sets it either). A panel Stop (StopTraining())
//--- still sets it and halts everything, including online learning - that distinction is
//--- preserved. See OnlineLearnStep()'s gate.
stop = true;
//--- Operator DELIBERATELY chose to deploy this best checkpoint as the final model. That's a
//--- terminal decision and must be PERSISTED as such: mark it complete so a later reload
//--- (chart restart OR strategy tester) runs inference instead of silently resuming a full
//--- training run. This is the terminal-deploy path; a mid-training Stop click
//--- (StopTraining()) leaves m_trainingComplete false on purpose so that genuinely-
//--- interrupted run does resume. Note the m_trainingComplete=(m_objectiveMet&&m_oosStable)
//--- line below is inside if(!stop), so it can't clobber this back to false on this path.
m_trainingComplete = true;
Print(ID + ": hit the " + IntegerToString(m_maxErasPerRun) + "-era cap before the plateau ladder finished (best dir-precision " + DoubleToString(m_bestBalancedOos, 1) + "%, blended OOS " + DoubleToString(dOosForecast, 1) + "%) - operator chose to DEPLOY the best checkpoint as final (marked complete; reloads will run inference, not retrain). Reaching this cap now means the run was still finding new bests, or never cleared the per-class recall floor (need >=" + IntegerToString(m_minDirectionalRecallPct) + "% each) - raise the era cap for the former, relax MinRecall/SwingConfirmationBars for the latter.");
}
}
}
if(!stop)
{
dError = Net.getRecentAverageError();
if(add_loop)
{
if(m_oosSamples > 0)
{
// Confidence calibration (classification head only - see m_confidenceCalScale's
// declaration comment): compare this era's actual OOS accuracy against the average
// confidence magnitude the model claimed, EMA-blend the resulting scale into
// m_confidenceCalScale so SignedAIConfidence() reports something closer to a real
// probability instead of the raw, uncalibrated softmax value.
if(m_outputNeuronsCount == 3 && m_oosConfidenceSum > 0.0)
{
double empiricalAccuracy = (double)(m_oosBuyHits + m_oosSellHits + m_oosNeutralHits) / m_oosSamples;
double avgClaimedConfidence = m_oosConfidenceSum / m_oosSamples;
double eraScale = MathMax(0.3, MathMin(1.5, empiricalAccuracy / avgClaimedConfidence));
m_confidenceCalScale += (eraScale - m_confidenceCalScale) / Net.recentAverageSmoothingFactor;
}
// Per-class recall gate, symmetric across all three classes: a model that "wins" on
// blended dOosForecast purely by calling everything Neutral (or, just as biased, by
// over-calling Buy/Sell at Neutral's expense) would still pass a plain accuracy check -
// require Buy, Sell, AND Neutral OOS recall to each individually clear
// m_minDirectionalRecallPct so the network can't converge while biased toward any one
// output. A class with FEWER than MIN_OOS_CLASS_SAMPLES_FOR_GATE true OOS samples this
// era doesn't block (recallPct == -1 => treated as passing) so a thin OOS window doesn't
// deadlock convergence early in a run. Computed BEFORE the checkpoint/eta-decay block
// below (not just the final m_objectiveMet gate) so "best" ranking is recall-aware too -
// see isBetterEra's comment for why that matters.
//
// The threshold matters: a bare ">0" here (the original behavior) let a run converge at
// era 44-46 with the OOS window containing exactly ZERO true Buy/Sell bars that era
// (logged as "OOS recall Buy:n/a Sell:n/a Neutral:100%") - a full Neutral-only collapse
// that the gate waved through because there was nothing to measure recall against, not
// because the model was actually unbiased. Requiring a real minimum sample count means
// an unlucky/thin OOS slice blocks convergence instead of silently passing it.
int buyRecallPct = (m_oosBuyTotal >= MIN_OOS_CLASS_SAMPLES_FOR_GATE) ? (int)MathRound(100.0 * m_oosBuyHits / m_oosBuyTotal) : -1;
int sellRecallPct = (m_oosSellTotal >= MIN_OOS_CLASS_SAMPLES_FOR_GATE) ? (int)MathRound(100.0 * m_oosSellHits / m_oosSellTotal) : -1;
int neutralRecallPct = (m_oosNeutralTotal >= MIN_OOS_CLASS_SAMPLES_FOR_GATE) ? (int)MathRound(100.0 * m_oosNeutralHits / m_oosNeutralTotal) : -1;
logBuyRecallPct = buyRecallPct;
logSellRecallPct = sellRecallPct;
logNeutralRecallPct = neutralRecallPct;
m_lastBuyRecallPct = buyRecallPct;
m_lastSellRecallPct = sellRecallPct;
// Predicted-rate (share of ALL OOS bars this era the model called this class, regardless
// of whether that call was right) and precision (of just those calls, how many were
// right) - see logBuyPredPct's declaration comment above for why this is worth logging
// alongside recall. Denominator is the per-era OOS bar count (sum of the per-class true
// totals, all tallied in the same pass-3 block and reset together each era) - NOT
// m_oosSamples, which only resets on a full model reset and so accumulates across every
// era of the run: dividing this era's calls by that all-run total diluted the logged
// rate by roughly the era number (observed: era-15 "Buy:2%" that was really ~30%),
// making a genuinely directional model read as a nearly-dead output.
int oosEraBars = m_oosBuyTotal + m_oosSellTotal + m_oosNeutralTotal;
logBuyPredPct = (oosEraBars > 0) ? (int)MathRound(100.0 * m_oosBuyPredicted / oosEraBars) : -1;
logSellPredPct = (oosEraBars > 0) ? (int)MathRound(100.0 * m_oosSellPredicted / oosEraBars) : -1;
logBuyPrecPct = (m_oosBuyPredicted > 0) ? (int)MathRound(100.0 * m_oosBuyPredictedHits / m_oosBuyPredicted) : -1;
logSellPrecPct = (m_oosSellPredicted > 0) ? (int)MathRound(100.0 * m_oosSellPredictedHits / m_oosSellPredicted) : -1;
//--- Live-fired precision (what actually trades - see m_oosBuyFired): of the directional
//--- calls that cleared the confidence floor under the live/prior-corrected rule this era,
//--- how many were right. Cached for the panel/log; -1 = the model fired none this era.
logBuyFiredPrecPct = (m_oosBuyFired > 0) ? (int)MathRound(100.0 * m_oosBuyFiredHits / m_oosBuyFired) : -1;
logSellFiredPrecPct = (m_oosSellFired > 0) ? (int)MathRound(100.0 * m_oosSellFiredHits / m_oosSellFired) : -1;
m_lastBuyFiredPrecPct = logBuyFiredPrecPct;
m_lastSellFiredPrecPct = logSellFiredPrecPct;
m_lastBuyFired = m_oosBuyFired;
m_lastSellFired = m_oosSellFired;
//--- SELECTION METRIC. Ranking moved off balanced accuracy (macro-recall) 2026-07-30
//--- because that metric is maximized by exactly the model this system must never deploy.
//--- Measured frontier at a fixed signal strength, base rate 6.1%:
//--- tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0%
//--- tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3%
//--- tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5%
//--- Balanced accuracy rises monotonically as the model calls MORE and is right LESS,
//--- because two of its three terms are directional recalls that a call-everything model
//--- drives to ~95% while the Neutral term it sacrifices counts for only a third. The
//--- 2026-07-29 run landed exactly there: balanced 58-64% while calling a direction on
//--- ~100% of bars at a 5-7% win rate against a ~6% base rate - no information at all.
//--- Only the per-class recall floor stopped those from deploying, i.e. a guard was doing
//--- the job the objective should have been doing, and the same guard also rejected the
//--- genuinely useful sparse-but-precise checkpoints (directional recall 4-6%).
//--- Ranking is now DIRECTIONAL PRECISION - of the bars this model called Buy or Sell,
//--- how many were right - which is what a trading edge actually is. Two anti-degenerate
//--- floors bracket it, because precision alone is trivially maximized by calling almost
//--- nothing: coverage must reach a fraction of the true base rate, and precision must at
//--- least beat that base rate (a model no better than the coin is not an edge).
int oosDirCalls = m_oosBuyPredicted + m_oosSellPredicted;
int oosDirHits = m_oosBuyPredictedHits + m_oosSellPredictedHits;
int oosDirTrue = m_oosBuyTotal + m_oosSellTotal;
bool coverageMeasurable = (oosEraBars > 0 && oosDirTrue > 0);
double coveragePct = coverageMeasurable ? 100.0 * oosDirCalls / oosEraBars : -1.0;
double baseRatePct = coverageMeasurable ? 100.0 * oosDirTrue / oosEraBars : -1.0;
double dirPrecPct = (oosDirCalls > 0) ? 100.0 * oosDirHits / oosDirCalls : -1.0;
double minCoveragePct = coverageMeasurable ? baseRatePct * MIN_COVERAGE_FRACTION_OF_BASE_RATE : -1.0;
//--- ZERO-SKILL PRECISION: what a model with no information scores on this metric, by
//--- always calling whichever direction is more common. Its precision is that class's
//--- share of ALL bars, because the bars it calls are uncorrelated with the labels.
//--- This REPLACED `dirPrecPct >= baseRatePct` on 2026-08-01, which was wrong the moment
//--- the labels stopped being rare. baseRatePct is Buy+Sell as a share of all bars: at the
//--- old exact-pivot target that was ~6%, so "beat the base rate" read as "beat chance"
//--- and the test looked sound. Triple-barrier labels put it at ~83%, and the gate then
//--- demanded 83% directional precision - unreachable by construction, so NOTHING could
//--- ever deploy. Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing
//--- safe to deploy" at a genuinely healthy 43-45% precision.
//--- max(Buy,Sell) is the right benchmark at ANY base rate: it is exactly the score of the
//--- degenerate always-call-one-direction model this floor exists to reject, and it
//--- degrades correctly to ~3% on the old rare-pivot labels.
double chancePrecPct = coverageMeasurable
? 100.0 * MathMax(m_oosBuyTotal, m_oosSellTotal) / oosEraBars : -1.0;
logCoveragePct = (int)MathRound(coveragePct);
logDirPrecPct = (int)MathRound(dirPrecPct);
logChancePrecPct = (chancePrecPct >= 0.0) ? (int)MathRound(chancePrecPct) : -1;
//--- Deployability. Replaces the per-class recall floor as the gate the checkpoint
//--- selection and the plateau ladder's "is there anything safe to deploy" test read.
//--- MinRecall still drives the diagnostic recall line below, but no longer decides what
//--- ships - it is the input that produced the catch-22 where nothing ever qualified.
//--- The margin is not arbitrary and not a knob: beating chance by any amount at all is a
//--- coin-flip result once the estimate's own sampling error is accounted for. With
//--- oosDirCalls directional calls at a chance rate p, the standard error of the measured
//--- precision is sqrt(p(1-p)/n) - about 0.4pp at the ~11,000 calls these runs produce - so
//--- `dirPrecPct > chancePrecPct` was passing models whose entire "edge" was under one
//--- sigma. Observed 2026-08-01: the perceptron deployed at edge +0pp.
//--- Requiring EDGE_MIN_SIGMAS standard errors instead scales the bar with the evidence:
//--- a sparse model needs a bigger measured edge to qualify than a dense one, which is
//--- exactly right, and no constant has to be re-tuned when coverage changes.
double chanceP = (chancePrecPct >= 0.0) ? chancePrecPct / 100.0 : 0.0;
double precSE = (oosDirCalls > 0 && chanceP > 0.0 && chanceP < 1.0)
? 100.0 * MathSqrt(chanceP * (1.0 - chanceP) / oosDirCalls) : 0.0;
double edgeFloorPct = chancePrecPct + EDGE_MIN_SIGMAS * precSE;
bool tradeableOK = coverageMeasurable && dirPrecPct >= 0.0 &&
coveragePct >= minCoveragePct && dirPrecPct > edgeFloorPct;
//--- Ranking key: precision, DISCOUNTED by how far short of the coverage floor the era
//--- fell. Raw precision was wrong here and the 2026-07-30 run caught it within 8 eras -
//--- HYBRID made exactly ONE directional call, got it right, scored 100%, and locked that
//--- in as best-ever. Nothing can beat 100%, so the checkpoint was frozen on a single
//--- sample and the run could only burn to the era cap. The coverage floor was already
//--- computed and already blocked that era from being DEPLOYABLE, but the ranking ignored
//--- it whenever no era had qualified yet - which is exactly the phase this matters in.
//--- Discounting rather than thresholding keeps the ordering continuous: an era at half
//--- the floor scores half its precision, so more coverage and better precision both
//--- improve rank and neither can be traded away entirely. Above the floor the credit
//--- saturates at 1.0, so ranking among genuinely deployable eras stays pure precision.
double coverageCredit = 1.0;
if(minCoveragePct > 0.0 && coveragePct >= 0.0)
coverageCredit = MathMin(1.0, coveragePct / minCoveragePct);
double selectionScore = (dirPrecPct >= 0.0) ? dirPrecPct * coverageCredit : 0.0;
bool directionalRecallOK = (buyRecallPct < 0 || buyRecallPct >= m_minDirectionalRecallPct) &&
(sellRecallPct < 0 || sellRecallPct >= m_minDirectionalRecallPct) &&
(neutralRecallPct < 0 || neutralRecallPct >= m_minDirectionalRecallPct);
// Balanced accuracy (macro-recall): the mean of the three per-class recalls - the metric
// the checkpoint SELECTION ranks on (see m_bestBalancedOos). Unlike blended accuracy it
// weights Buy, Sell and Neutral equally, so it can't be inflated by the ~96%-Neutral base
// rate. Computed from the same raw per-era recalls the floor uses (not smoothed - the
// whole recall-driven side of this block is per-era-raw by design). When a directional
// class is thin/unmeasured this era (recall -1), balanced accuracy isn't meaningful, so
// fall back to the blended dOosForecast for ranking that era (prior behavior) rather than
// averaging a partial set - the recall floor + directionalRecallMeasured still guard the
// actual convergence decision separately.
double balancedOosEra = (buyRecallPct >= 0 && sellRecallPct >= 0 && neutralRecallPct >= 0)
? (buyRecallPct + sellRecallPct + neutralRecallPct) / 3.0
: dOosForecast;
logBalancedAccPct = (buyRecallPct >= 0 && sellRecallPct >= 0 && neutralRecallPct >= 0)
? (int)MathRound(balancedOosEra) : -1;
// A real (non-thin-sample, i.e. not the -1 "n/a" sentinel) 0% recall on any class means
// the model never once got that class right this era - a majority-class collapse
// (predict-everything-Neutral, or symmetrically a Buy/Sell-only collapse), not progress
// toward separating classes. Before any era has ever passed the recall floor,
// isBetterEra's fallback below is a pure blended-accuracy tiebreak, and blended accuracy
// is trivially maximized by collapsing to the majority class. Observed in practice
// (2026-07-19, SP500 H4): once a run landed on a 0%/0%/100% Buy/Sell/Neutral era, its
// accuracy kept creeping upward for 124 STRAIGHT eras purely from sharpening the
// Neutral-vs-everything boundary - each tick registered as a "new best", re-anchoring the
// checkpoint AND bumping eta back toward its ceiling (the recovery bump below), actively
// rewarding the collapse instead of remaining neutral to it. Excluding these eras from
// isBetterEra denies them that anchor/reward without touching the restore/decay branch
// below, which stays exactly as gated on m_bestPassedRecall as before - see that block's
// own comment for why loosening THAT part pre-pass caused a worse failure historically.
//--- Under precision ranking the degenerate era is the one that called NOTHING
//--- directional (precision undefined, nothing to trade), not one whose per-class
//--- recall touched zero - a sparse high-precision model legitimately has low recall.
bool isFullyCollapsedEra = (oosDirCalls <= 0);
// Lexicographic "better than the best-so-far" ordering: passing the directional recall
// floor always outranks not passing it, regardless of blended dOosForecast; only WITHIN
// the same pass/fail category does blended accuracy break the tie. Without this, an era
// that traded a few "safe" Neutral calls for genuinely useful (recall-improving) Buy/Sell
// calls would look like a regression in blended-accuracy-only terms and get its
// checkpoint skipped / learning rate cut - fighting directly against the network learning
// to call Buy/Sell at all, since Neutral is the large majority class (~80%+ of labels) and
// a model that just calls everything Neutral already scores well on blended accuracy
// alone. isWorseEra mirrors the same ordering for the eta-decay-on-regression trigger.
// (directionalRecallOK implies !isFullyCollapsedEra already, since the floor is always
// >0%, so the first clause below needs no extra guard - only the pre-pass accuracy-only
// tiebreak in the second clause does.)
// Within the same recall-pass category the tie now breaks on BALANCED accuracy, not the
// Neutral-dominated blended dOosForecast - see m_bestBalancedOos. This is what deploys the
// most class-balanced era instead of the most Neutral-leaning one, and it also strengthens
// the pre-pass phase: a Neutral-only era scores (0+0+N)/3 in balanced terms (low) rather
// than the ~80% it scores in blended terms, so it can no longer re-anchor the checkpoint.
//--- tradeableOK / selectionScore, not directionalRecallOK / balancedOosEra - see the
//--- SELECTION METRIC note above. The lexicographic shape is unchanged: qualifying
//--- always outranks not qualifying, and the score only breaks ties within a category.
bool isBetterEra = (tradeableOK && !m_bestPassedRecall) ||
(tradeableOK == m_bestPassedRecall && !isFullyCollapsedEra && selectionScore > m_bestBalancedOos);
// The recall-pass-loss clause used to fire on ANY drop out of a full 3-way recall pass,
// even a near-miss on one class at unchanged accuracy (e.g. observed: Buy:56% Sell:41%
// Neutral:34% - Neutral alone missing the 40% floor by a few points) - treating that
// identically to a total collapse back to Neutral-only. With three classes all needing
// to simultaneously clear the floor, that made isWorseEra fire on most eras once a pass
// was ever achieved, ratcheting eta toward ETA_MIN within a handful of eras and then
// (before the recovery bump below existed) leaving it stuck there permanently - visible
// in practice as ~25 back-to-back identical "regressed from best 70.6% to 70.6%" eras.
// Now only counts as worse if accuracy ALSO dropped meaningfully (same threshold
// regardless of whether the recall-pass flag changed too) - losing the recall-pass flag
// at flat/improved accuracy is borderline variance, not a regression worth
// restoring+decaying over.
bool isWorseEra = selectionScore < m_bestBalancedOos - ETA_DECAY_REGRESSION_PCT;
if(isBetterEra)
{
//--- Snapshot BOTH scores at the checkpoint: m_bestBalancedOos is what ranking compares
//--- against next era; m_bestOosForecast keeps the blended value FinalizeTrainRun() and
//--- the restore branch reset dOosForecast to (see m_bestBalancedOos' declaration).
m_bestOosForecast = dOosForecast;
m_bestBalancedOos = selectionScore;
m_bestPassedRecall = tradeableOK;
//--- eval candidates are throwaway - track the score (above) but never write a checkpoint
//--- file; m_haveOosCheckpoint=false then also skips the worse-era RestoreWeights() restore.
//--- In-MEMORY weight snapshot (not a file): the file-based checkpoint re-created every
//--- neuron on restore, which the CPU-DLL backend can't do for a second live set - see
//--- CNet::CaptureWeights/RestoreWeights. eval candidates snapshot nothing.
m_haveOosCheckpoint = Net.CaptureWeights();
// Recovery bump: ETA_DECAY_FACTOR-only ever shrinks eta, and previously nothing ever
// grew it back - a losing streak early in a run (even a since-corrected one) would
// permanently cap how fast every later era could learn for the rest of the run, all
// the way down to ETA_MIN with no way back. A genuinely better era (new best, not
// just a tie) means the current eta is working, so nudge it back up a bit - capped at
// this model's own configured starting rate (m_etaCeiling - AdamLearningRate for
// ADAM, SgdLearningRate for SGD, see that member's declaration comment) so this
// can't runaway past the rate training was actually tuned to start at.
eta = MathMin(m_etaCeiling, eta / ETA_DECAY_FACTOR);
}
else
if(isWorseEra && m_bestOosForecast > 0)
{
// Decaying eta alone only softens FUTURE steps - it does nothing to undo the
// regression this era already baked into the weights, so a run could (and in
// practice did) spend 15+ eras compounding forward from one bad era's damage,
// each new era fighting the last one's overshoot instead of building on the best
// state found so far. Restore the last checkpointed-good weights before continuing
// (mirrors what FinalizeTrainRun() does at the END of a run, just applied live so
// the oscillation can't compound within a single run) - this is what actually turns
// "reduce LR on regression" into "step back, then retry slower", not just "drift
// slower".
//
// BOTH the restore AND the eta decay below are gated on m_bestPassedRecall: before
// ANY era has ever cleared the per-class recall floor, isBetterEra's own
// lexicographic ordering degrades to a pure blended-accuracy tiebreak
// (directionalRecallOK==false on both sides of the comparison), so "best checkpoint"
// during that phase just means "called Neutral most confidently so far" - restoring
// it would actively defend the majority-class collapse against any era that trades
// some accuracy for real Buy/Sell recall, which is exactly the bias this whole
// recall-gate mechanism exists to prevent (see isBetterEra's own comment above).
// Observed in practice: era 1-3 all "improved" on accuracy alone
// (24.9%->41.4%->52.3%) while Buy/Sell recall stayed at a flat 0% the entire time -
// restoring pre-pass would have locked training into that trajectory instead of
// letting it explore past it. Decaying eta has the same bias one step removed:
// every regression relative to a Neutral-collapse "best" shrinks eta a little more,
// steadily strangling the exploration needed to escape that collapse until eta
// bottoms out at ETA_MIN with no real solution ever found and no checkpoint to fall
// back on either - observed in practice as a run whose best-ever blended accuracy
// kept landing on 0%/0%/100% Buy/Sell/Neutral recall eras, each one triggering
// another decay on the very next era, until eta floored out around era 20 and the
// remaining eras just oscillated between collapse states with no way to make a
// large-enough move to escape and no way to reset. Once m_bestPassedRecall is true,
// there IS a genuinely good state worth protecting, and both restoring the
// checkpoint and decaying eta on regression are safe/correct again.
// 2026-07-29: the m_bestPassedRecall gate above has an escape now, because its
// stated premise expired. It was written when the pre-pass tiebreak really was
// blended-accuracy-only; the balanced-selection change (m_bestBalancedOos) replaced
// that with `balancedOosEra > m_bestBalancedOos` AND an isFullyCollapsedEra
// exclusion, so a Neutral-only era now scores ~33% (the FLOOR of the balanced
// metric) and cannot anchor the checkpoint at all. "Best checkpoint" pre-pass
// therefore no longer means "called Neutral most confidently" - it means "most
// class-balanced state found so far", which is worth defending, and isWorseEra is
// itself a balanced-accuracy regression, so it cannot fire merely for trading
// Neutral calls for Buy/Sell.
//
// Leaving the gate absolute had a failure mode of its own, and it is not
// hypothetical: if NO checkpoint ever clears the recall floor, m_bestPassedRecall
// stays false forever, so there is never any restore and never any eta decay.
// Observed on SP500 H1 2026-07-29 across three topologies - CONV ran 228 eras with
// eta pinned at its 0.000300 start while balanced accuracy slid 40% -> 35% and Buy
// recall 11% -> 2%. The run had no regression control whatsoever, and the plateau
// ladder could not end it either (stage 3 refuses to deploy without a recall pass),
// so it was a 1000-era one-way trip into a Neutral collapse.
//
// The original concern still applies while the best-so-far IS near-collapse:
// decaying eta against such a "best" strangles the exploration needed to escape it.
// So the escape is margin-guarded - defend the checkpoint only once it sits clearly
// above the one-class floor, which is exactly when there is something real to lose.
bool bestWorthDefending = (m_bestBalancedOos >
BALANCED_COLLAPSE_PCT + BALANCED_WORTH_DEFENDING_MARGIN_PCT);
if(m_bestPassedRecall || bestWorthDefending)
{
if(m_haveOosCheckpoint && Net.RestoreWeights())
dOosForecast = m_bestOosForecast;
if(eta > ETA_MIN)
eta = MathMax(ETA_MIN, eta * ETA_DECAY_FACTOR);
Print(ID + ": OOS selection score (coverage-weighted dir-precision) regressed from best " + DoubleToString(m_bestBalancedOos, 1) +
"% to " + DoubleToString(selectionScore, 1) + "% (blended " + DoubleToString(m_bestOosForecast, 1) +
"%->" + DoubleToString(dOosForecast, 1) + "%) - restoring best checkpoint and decaying learning rate to " + DoubleToString(eta, 6));
}
else
Print(ID + ": OOS selection score (coverage-weighted dir-precision) regressed from best " + DoubleToString(m_bestBalancedOos, 1) +
"% to " + DoubleToString(selectionScore, 1) + "% (blended " + DoubleToString(m_bestOosForecast, 1) +
"%->" + DoubleToString(dOosForecast, 1) + "%) - best so far is still within " +
DoubleToString(BALANCED_WORTH_DEFENDING_MARGIN_PCT, 1) + "pp of the " +
DoubleToString(BALANCED_COLLAPSE_PCT, 1) + "% one-class floor, so there is nothing worth" +
" restoring yet - continuing to explore without decaying eta (still " + DoubleToString(eta, 6) + ")");
}
//=== PLATEAU LADDER ====================================================================
//--- Neither branch above fires in the dead zone between "new best" and "regressed by more
//--- than ETA_DECAY_REGRESSION_PCT". This is the response to sitting in it: count eras since
//--- the last new best and escalate. See the PLATEAU_* constants for the full rationale and
if(isBetterEra)
{
//--- Moving again: retire the ladder. Deliberately does NOT undo the warm restart's
//--- raised eta - if the bigger step is what produced this new best, decaying it back
//--- would undo the very change that worked. The normal per-era eta schedule takes
//--- over from here.
if(m_plateauStage > 0)
Print(ID + ": new best selection score (coverage-weighted dir-precision) " + DoubleToString(m_bestBalancedOos, 1) +
"% - plateau escape worked, clearing plateau stage " + IntegerToString(m_plateauStage));
m_erasSinceBestBalanced = 0;
m_plateauStage = 0;
}
else
{
m_erasSinceBestBalanced++;
int dueStage = m_erasSinceBestBalanced / PLATEAU_PATIENCE_ERAS;
if(dueStage > m_plateauStage)
{
m_plateauStage = dueStage;
string stageNote = IntegerToString(m_erasSinceBestBalanced) + " eras with no new best selection score (best " +
DoubleToString(m_bestBalancedOos, 1) + "%)";
if(m_plateauStage == PLATEAU_STAGE_RESTART || m_plateauStage == PLATEAU_STAGE_ANNEAL)
{
//--- WARM RESTART: jump eta back to this model's configured starting rate. A
//--- plateau needs a bigger step to climb out of its basin, not a smaller one.
double etaBefore = eta;
eta = m_etaCeiling;
//--- The focal-gamma anneal that used to accompany this went with focal loss
//--- on 2026-07-31. It was only ever a monotone step toward zero on a second
//--- imbalance correction; the warm restart above is and always was the
//--- actual escape, so both ladder stages keep their distinct patience
//--- thresholds and simply retry the restart.
Print(ID + ": PLATEAU stage " + IntegerToString(m_plateauStage) + " - " + stageNote +
". Warm restart: learning rate " + DoubleToString(etaBefore, 6) + "->" + DoubleToString(eta, 6) +
". Best checkpoint is safe - this only changes how the NEXT eras train.");
}
else
if(m_plateauStage >= PLATEAU_STAGE_DEPLOY)
{
//--- Exhausted: both escapes were tried and neither found a better model, so
//--- this IS the best this configuration reaches. The deploy itself happens in
//--- the era-cap/plateau branch at the TOP of the next era, which reuses the
//--- proven "stop + mark complete -> FinalizeTrainRun restores and deploys the
//--- best checkpoint" path rather than duplicating it here.
//--- Safety: only ever auto-deploys a checkpoint that CLEARED the per-class
//--- recall floor (m_bestPassedRecall). If nothing ever did, there is no model
//--- worth deploying - so the ladder resets and keeps trying instead, leaving
//--- the era cap as the ultimate backstop. That is what stops "train to the
//--- best possible result" from degenerating into "deploy a Neutral collapse".
if(m_bestPassedRecall && m_haveOosCheckpoint)
Print(ID + ": PLATEAU stage " + IntegerToString(PLATEAU_STAGE_DEPLOY) + " - " + stageNote +
" across " + IntegerToString(PLATEAU_STAGE_DEPLOY - 1) + " warm restarts. Training has converged on what this"
+ " configuration can reach - deploying the best checkpoint (dir-precision "
+ DoubleToString(m_bestBalancedOos, 1) + "%, blended " + DoubleToString(m_bestOosForecast, 1) + "%).");
else
{
Print(ID + ": PLATEAU stage " + IntegerToString(PLATEAU_STAGE_DEPLOY) + " - " + stageNote +
", but no checkpoint has ever cleared the deployability floor (directional calls on" +
" at least a quarter as many bars as actually swing, at a precision above that base rate), so there is nothing safe to"
+ " deploy. Restarting the plateau ladder and continuing to train rather than deploying a"
+ " one-class model; the " + IntegerToString(m_maxErasPerRun) + "-era cap remains the backstop.");
m_erasSinceBestBalanced = 0;
m_plateauStage = 0;
}
}
}
}
m_oosWindow.Add(dOosForecast);
while(m_oosWindow.Total() > STABILITY_WINDOW)
m_oosWindow.Delete(0);
m_oosStable = false;
if(m_oosWindow.Total() >= STABILITY_WINDOW)
{
double oosMin = m_oosWindow.At(0), oosMax = m_oosWindow.At(0);
for(int w = 1; w < m_oosWindow.Total(); w++)
{
oosMin = MathMin(oosMin, m_oosWindow.At(w));
oosMax = MathMax(oosMax, m_oosWindow.At(w));
}
m_oosStable = (oosMax - oosMin) <= STABILITY_TOLERANCE;
}
// The dError<0.1 RMS-error floor is meaningful for the single-neuron regression head
// (m_outputNeuronsCount==1), where it's the only convergence signal available. For the
// 3-neuron one-hot classification head it's redundant with, and far stricter than,
// dOosForecast/directionalRecallOK: reaching RMS error 0.1 across 3 one-hot targets
// needs every output neuron within ~0.17 of its target on average, i.e. near-perfect
// confident calibration on EVERY bar, not just correct argmax calls - unreachable in
// practice under normal market label noise, so classification runs would oscillate
// forever (era after era hitting good OOS accuracy and passing recall, but never
// satisfying this) without this carve-out.
bool errorGateOK = (m_outputNeuronsCount == 3) ? true : (dError < 0.1);
// Convergence (unlike isBetterEra's ranking) FINALIZES the model, so both directional
// classes must have actually been MEASURED this era. An n/a (-1, thin-sample) Buy or
// Sell recall passing directionalRecallOK is deliberate for ranking (early thin
// windows shouldn't deadlock "best" tracking), but letting it pass HERE converges on
// a window that contained no directional bars to disprove the model. Observed
// 2026-07-19: a mid-run label-cache wipe relabeled the whole window Neutral,
// "accuracy" hit 84.9% with Buy/Sell recall both n/a - without this gate a Neutral-only
// model finalizes as a certified success.
bool directionalRecallMeasured = (m_outputNeuronsCount != 3) || (buyRecallPct >= 0 && sellRecallPct >= 0);
//--- VALIDITY of this era's model, no longer "did it hit a target accuracy". The absolute
//--- OOS-accuracy target (the old MinWR input) is gone: an accuracy number typed in ahead of
//--- time is either unreachable for the symbol/timeframe - in which case the run never
//--- converges and burns to the era cap - or set low enough to stop a run that was still
//--- improving. Neither is what "train to the best result" means. Quality is now enforced by
//--- WHICH era gets deployed (balanced-accuracy checkpoint ranking + this per-class recall
//--- floor) and WHEN a run ends (the plateau ladder), not by an accuracy threshold. Note the
//--- recall floor deliberately stays: it is not a performance target but the anti-collapse
//--- gate that makes auto-deploy safe.
m_objectiveMet = errorGateOK && directionalRecallOK && directionalRecallMeasured;
}
// Only mark the persisted model "complete" once it actually converged this era -
// an interruption (stop) or an ordinary in-progress era must stay flagged incomplete
// so a restart resumes training instead of quietly treating a partial run as done.
// Convergence = "the plateau ladder is exhausted AND there is a recall-passing checkpoint
// to deploy" - the exact same condition the deploy branch beside the era-cap check uses, so
// the flag written into the .nnw here can never disagree with the decision to stop. While a
// run is still improving (or still has an escape stage left to try) this stays false and the
// per-era save correctly records an in-progress run. Previously this was
// (m_objectiveMet && m_oosStable), which needed the removed absolute accuracy target to mean
// anything: with that target gone m_oosStable alone - just 3 eras inside a 2pp band, which
// is true constantly - would have converged the run at the first flat spot.
m_trainingComplete = (m_plateauStage >= PLATEAU_STAGE_DEPLOY) && m_bestPassedRecall && m_haveOosCheckpoint;
double currentIndicatorParams[];
m_indicatorTuner.Flatten(currentIndicatorParams);
if(!Net.Save(m_activeFileName + ".nnw", dError, dUndefine, dForecast, dtStudied, m_activeFileCommon, m_eraCount, m_trainingComplete, currentIndicatorParams))
Print(__FUNCTION__ + ": ERROR - era-end Net.Save failed for " + m_activeFileName + ".nnw (era " + IntegerToString(m_eraCount) + "). Training continues but this era's checkpoint was NOT persisted - a crash/restart now would resume from an older era.");
if(!SaveModelStats(m_activeFileName, m_activeFileCommon)) // keep calibration state paired with the just-saved weights
Print(__FUNCTION__ + ": ERROR - SaveModelStats failed for " + m_activeFileName + " (era " + IntegerToString(m_eraCount) + "). Calibration/online-learning state not persisted this era.");
SaveShadowNet(currentIndicatorParams);
}
}
if(shouldLogProgress)
{
string recallInfo = (logBuyRecallPct < 0 && logSellRecallPct < 0 && logNeutralRecallPct < 0) ? "" :
(" | OOS recall Buy:" + (logBuyRecallPct < 0 ? "n/a" : IntegerToString(logBuyRecallPct) + "%") +
" Sell:" + (logSellRecallPct < 0 ? "n/a" : IntegerToString(logSellRecallPct) + "%") +
" Neutral:" + (logNeutralRecallPct < 0 ? "n/a" : IntegerToString(logNeutralRecallPct) + "%") +
" (need >=" + IntegerToString(m_minDirectionalRecallPct) + "% each)");
//--- Balanced accuracy = the checkpoint-selection metric (see m_bestBalancedOos). Shown so the
//--- number the deployed model is actually chosen on is visible next to the recalls it averages.
//--- Balanced accuracy is retained as a DIAGNOSTIC only - selection ranks on directional
//--- precision now (see the SELECTION METRIC note). Both are shown so a run where they
//--- disagree - the signature of an over-calling model - is visible at a glance.
string balancedInfo = (logBalancedAccPct < 0) ? "" : (" | OOS balanced acc " + IntegerToString(logBalancedAccPct) + "% (diagnostic)");
string selectionInfo = (logDirPrecPct < 0) ? " | SELECT: no directional calls" :
(" | SELECT dir-precision " + IntegerToString(logDirPrecPct) + "% on " +
IntegerToString(logCoveragePct) + "% of bars" +
(logChancePrecPct >= 0
? " (chance " + IntegerToString(logChancePrecPct) + "%, edge " +
(logDirPrecPct - logChancePrecPct >= 0 ? "+" : "") +
IntegerToString(logDirPrecPct - logChancePrecPct) + "pp)"
: ""));
// See logBuyPredPct's declaration comment for why this is worth logging alongside recall -
// it's what tells apart a suppressed/dead output (predicted rate stuck at 0%) from a
// miscalibrated boundary (predicted rate healthy, precision poor), which look identical from
// recall alone.
string predictedInfo = (logBuyPredPct < 0 && logSellPredPct < 0) ? "" :
(" | OOS calls Buy:" + (logBuyPredPct < 0 ? "n/a" : IntegerToString(logBuyPredPct) + "%") +
" (win rate " + (logBuyPrecPct < 0 ? "n/a" : IntegerToString(logBuyPrecPct) + "%") + ")" +
" Sell:" + (logSellPredPct < 0 ? "n/a" : IntegerToString(logSellPredPct) + "%") +
" (win rate " + (logSellPrecPct < 0 ? "n/a" : IntegerToString(logSellPrecPct) + "%") + ")");
//--- Live-fired precision: the number that actually predicts forward-trading performance - only
//--- the directional calls that cleared the confidence floor under the live/prior-corrected rule
//--- (see AdjustedSignalFromSoftmax). Count in parentheses = how many bars the model would have
//--- traded this era. "0" fires = the calibration is (this era) suppressing all directional trades.
string liveInfo = (m_lastBuyFired <= 0 && m_lastSellFired <= 0) ? " | live fires 0 this era" :
(" | live win rate Buy:" + (logBuyFiredPrecPct < 0 ? "n/a" : IntegerToString(logBuyFiredPrecPct) + "%") +
" (" + IntegerToString(m_lastBuyFired) + ")" +
" Sell:" + (logSellFiredPrecPct < 0 ? "n/a" : IntegerToString(logSellFiredPrecPct) + "%") +
" (" + IntegerToString(m_lastSellFired) + ")");
//--- Precision BY CONFIDENCE TIER, and cumulatively from each tier upward - the two numbers a
//--- decision about Min_Vote_Open actually needs. The per-tier figure says whether confidence is
//--- calibrated to correctness at all (it should rise T0->T3; if it does not, raising the floor
//--- buys nothing and the finding is that the head's confidence is uninformative). The ">=Tn"
//--- figure is what you would ACTUALLY get, because a floor keeps every tier at or above it, and
//--- it comes with the fire count so the coverage cost of raising the floor is visible in the
//--- same line. Tier weights are 25/50/75/100, so for an AI-only config the input maps straight
//--- across: Min_Vote_Open 50 = ">=T1", 75 = ">=T2", 100 = ">=T3".
string tierInfo = "";
int tierFiredTotal = 0;
for(int ti = 0; ti < 4; ti++)
tierFiredTotal += m_oosTierFired[ti];
if(tierFiredTotal > 0)
{
tierInfo = " | tier prec";
for(int ti = 0; ti < 4; ti++)
{
int cumFired = 0, cumHits = 0;
for(int tj = ti; tj < 4; tj++)
{
cumFired += m_oosTierFired[tj];
cumHits += m_oosTierHits[tj];
}
tierInfo += " T" + IntegerToString(ti) + ":" +
(m_oosTierFired[ti] > 0
? IntegerToString((int)MathRound(100.0 * m_oosTierHits[ti] / m_oosTierFired[ti])) + "%"
: "n/a") +
"(" + IntegerToString(m_oosTierFired[ti]) + ")" +
(cumFired > 0
? "[>=" + IntegerToString((int)MathRound(100.0 * cumHits / cumFired)) + "%/" +
IntegerToString(cumFired) + "]"
: "");
}
}
// Per-layer weight movement. Pairs with rawOutInfo below: a collapsed constant-classifier state
// has two very different causes, and only this tells them apart. If every layer moves and the
// output still collapses, the architecture or the objective is at fault; if one stage sits at
// ~0.000% era after era while the others move, that stage is receiving no gradient and no amount
// of retraining or hyperparameter work will help. See CNet::LayerLearningReport.
string layerInfo = (CheckPointer(Net) == POINTER_INVALID) ? "" :
(" | dW/W" + Net.LayerLearningReport());
// Raw-output saturation diagnostic - see m_oosOutMin's declaration comment. Spread ~0 with
// all six min/max values pinned together = the collapsed constant-classifier state.
string rawOutInfo = (m_oosOutCount <= 0) ? "" :
StringFormat(" | OOS raw out B:%.3f..%.3f S:%.3f..%.3f N:%.3f..%.3f spread avg %.4f",
m_oosOutMin[0], m_oosOutMax[0], m_oosOutMin[1], m_oosOutMax[1],
m_oosOutMin[2], m_oosOutMax[2], m_oosOutSpreadSum / m_oosOutCount);
//--- No "(target X%)" any more - there is no absolute accuracy target. What replaces it as the
//--- progress indicator is the plateau counter: how many eras since the last new best, and how
//--- close that is to ending the run (see the PLATEAU_* ladder).
string plateauInfo = (m_bestBalancedOos < 0) ? "" :
(" | best bal " + DoubleToString(m_bestBalancedOos, 1) + "%, " + IntegerToString(m_erasSinceBestBalanced) +
" eras since (stage " + IntegerToString(m_plateauStage) + "/" + IntegerToString(PLATEAU_STAGE_DEPLOY) + ")");
//--- Lifetime IS/OOS directional accuracy. The panel now shows the out-of-sample half alone (see
//--- ComputeCompoundedAccuracyLine - the in-sample figure grades the model on bars it trained on,
//--- so it always reads higher than anything forward trading will deliver and does not belong on
//--- a product's face). The GAP between the two is still the over-fitting read, so it survives
//--- here, once per era, behind the compile-time DebuggingMode constant.
string lifetimeInfo = (!DebuggingMode || (m_cumIsTotal <= 0 && m_cumOosTotal <= 0)) ? "" :
(" | lifetime dir acc IS " + (m_cumIsTotal > 0 ? IntegerToString((int)MathRound(m_cumIsCorrect * 100.0 / m_cumIsTotal)) + "%" : "n/a") +
" OOS " + (m_cumOosTotal > 0 ? IntegerToString((int)MathRound(m_cumOosCorrect * 100.0 / m_cumOosTotal)) + "%" : "n/a") +
" over " + IntegerToString(m_cumIsTotal + m_cumOosTotal) + " calls");
Print(ID + ": training in progress - era " + IntegerToString(m_eraCount) + ", OOS accuracy " + DoubleToString(dOosForecast, 1) + "%, IS error " + DoubleToString(dError, 2) + recallInfo + balancedInfo + selectionInfo + predictedInfo + liveInfo + tierInfo + plateauInfo + lifetimeInfo + rawOutInfo + layerInfo);
// Forced (unthrottled) panel refresh, right here alongside the console line above, using this
// era's own just-finalized m_eraCount/dOosForecast - see UpdateTrainingStatusLabel's
// declaration comment for why this can't just rely on the next throttled bar-scan call to
// catch up (it would, but a full era later than the console already reported it).
UpdateTrainingStatusLabel("Era complete", m_lastDisplayNeuron0, m_lastDisplayNeuron1, m_lastDisplayNeuron2, m_lastDisplaySignal, true);
}
//--- Genuine convergence THIS era (not a stale m_trainingComplete carried over from a previous
//--- run) - (re)start the evaluation-only continual-learning OOS walk. Always rebuilt fresh from
//--- the just-converged weights; never resumes a stale walk from a superseded model.
//--- m_trainingComplete is the plateau ladder's verdict now (see where it is assigned): "stopped
//--- improving after both escape attempts, and there is a recall-passing checkpoint to deploy".
//--- It replaces the old (m_objectiveMet && m_oosStable) test, which depended on the removed
//--- absolute accuracy target to mean anything - without it, m_oosStable alone (3 eras inside a 2pp
//--- band) would have declared convergence at the first flat spot in every run.
if(!stop && m_trainingComplete)
{
Print(ID + ": training CONVERGED at era " + IntegerToString(m_eraCount) + " - this is the best this configuration reached: dir-precision " +
DoubleToString(m_bestBalancedOos, 1) + "%, blended OOS " + DoubleToString(dOosForecast, 1) + "%, IS error " + DoubleToString(dError, 2) +
". No new best for " + IntegerToString(m_erasSinceBestBalanced) + " eras across " +
IntegerToString(PLATEAU_STAGE_DEPLOY - 1) + " learning-rate warm restarts." +
" Weights saved, switching to live inference.");
StartOosContinualSimulation(bars, oosCutoff);
}
if(stop || m_trainingComplete)
FinalizeTrainRun();
//--- else: this era is done but the run continues - the next Train() call (re-triggered via
//--- ScheduleTrainingIfNeeded()'s custom event, same mechanism as always) starts the next era
//--- fresh, since m_eraResumePending is false while m_trainRunActive stays true
//--- Save this model's own learning-rate trajectory back out of the shared global before
//--- returning - see m_modelEta's declaration comment. Covers every path that reaches here
//--- (natural era completion, whether or not the run itself just finalized).
m_modelEta = eta;
}
//+------------------------------------------------------------------+
//| Ends the current Train() run: restores the best-scoring era's |
//| checkpointed weights (if any beat the era the loop happened to |
//| end on), persists final state, and clears the resumable-run |
//| flags. Called both from Train() itself (natural stop/converge) |
//| and from StopTraining() (a mid-chunk Stop click won't get |
//| another "New Bar" event to resume into, since |
//| ScheduleTrainingIfNeeded() refuses to schedule while |
//| m_trainingStopRequested is set, so it must finalize synchronously |
//| there instead of being left dangling). |
//+------------------------------------------------------------------+
//| Era-cap decision: keep training (true) or deploy best + stop |
//| (false). Live chart -> operator dialog; headless -> stop. |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::PromptContinuePastEraCap(double bestOos)
{
//--- No GUI in the Strategy Tester/optimizer - MessageBox() is unavailable there and would just
//--- stall a headless run, so deploy the best checkpoint found so far and stop (the safe default).
if(MQLInfoInteger(MQL_TESTER) || MQLInfoInteger(MQL_OPTIMIZATION) || MQLInfoInteger(MQL_FORWARD))
return false;
//--- Reaching this cap is now the UNUSUAL outcome: a run normally ends itself when the plateau ladder
//--- runs out of escapes (see the PLATEAU_* constants), which is a statement about the run having
//--- stopped improving rather than about any accuracy number. So the interesting question here is why
//--- the ladder had not finished yet - either the run was still finding new bests (just needs more
//--- eras), or nothing has ever cleared the per-class recall floor, which blocks auto-deploy on
//--- purpose so a one-class model can never ship. Spell out which.
bool recallMet = (m_lastBuyRecallPct < 0 || m_lastBuyRecallPct >= m_minDirectionalRecallPct) &&
(m_lastSellRecallPct < 0 || m_lastSellRecallPct >= m_minDirectionalRecallPct);
string neutralNote = (m_priorNeutral > 0.0)
? ("inflated by the ~" + IntegerToString((int)MathRound(m_priorNeutral * 100.0)) + "% Neutral base rate")
: "inflated by the dominant Neutral class";
string reasons = "";
if(!m_bestPassedRecall)
reasons += " - No era has ever cleared the per-class recall floor, so there is no model safe to\n" +
" auto-deploy yet (a model that ignores Buy or Sell must never ship)\n";
else
reasons += " - Still improving: " + IntegerToString(m_erasSinceBestBalanced) + " eras since the last new best, plateau stage " +
IntegerToString(m_plateauStage) + " of " + IntegerToString(PLATEAU_STAGE_DEPLOY) + " (the run ends itself at stage " +
IntegerToString(PLATEAU_STAGE_DEPLOY) + ")\n";
if(!recallMet)
reasons += " - Latest era's per-class recall below the floor: Buy " +
(m_lastBuyRecallPct < 0 ? "n/a" : IntegerToString(m_lastBuyRecallPct) + "%") + " / Sell " +
(m_lastSellRecallPct < 0 ? "n/a" : IntegerToString(m_lastSellRecallPct) + "%") +
" (need >=" + IntegerToString(m_minDirectionalRecallPct) + "% each)\n";
if(!m_objectiveMet)
reasons += " - The latest era did not produce a valid model (recall floor not met/not measured)\n";
string balancedStr = (m_bestBalancedOos > 0.0)
? ("\nBest directional precision, coverage-weighted (the metric the deployed\ncheckpoint is chosen on): " + DoubleToString(m_bestBalancedOos, 1) +
"%\nBest blended OOS accuracy: " + DoubleToString(bestOos, 1) + "% (" + neutralNote + ")\n")
: "";
string msg = ID + ": training reached the " + IntegerToString(m_maxErasPerRun) +
"-era cap before it finished on its own.\n\n" +
"Training now runs until it stops improving, then deploys its best model. Status:\n" +
reasons +
balancedStr +
"\nContinue training?\n\n" +
"Yes = keep training for another " + IntegerToString(m_maxErasPerRun) + " eras\n" +
"No = deploy the best checkpoint so far and stop training";
int res = MessageBox(msg, "Warrior EA - training", MB_YESNO | MB_ICONQUESTION);
return (res == IDYES);
}
//+------------------------------------------------------------------+
//+------------------------------------------------------------------+
//| See the declaration comment - the single deploy-persistence path. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::PersistDeployedModel(void)
{
if(CheckPointer(Net) == POINTER_INVALID)
return;
double currentIndicatorParams[];
m_indicatorTuner.Flatten(currentIndicatorParams);
if(!Net.Save(m_activeFileName + ".nnw", dError, dUndefine, dForecast, dtStudied, m_activeFileCommon, m_eraCount, m_trainingComplete, currentIndicatorParams))
Print(__FUNCTION__ + ": ERROR - Net.Save failed for " + m_activeFileName + ".nnw. The deployed model was NOT persisted to disk.");
//--- Deploy-time gate: does this model's pure-MQL5 forward pass match the backend? If so, an
//--- inference-only backtest can run DLL-free (see ValidateCpuInference / CNet::SetCpuInference).
//--- Persisted into the .stats written next. Chart-only; safe-false everywhere else.
m_mqlInferenceValidated = ValidateCpuInference();
if(!SaveModelStats(m_activeFileName, m_activeFileCommon)) // keep calibration state paired with the just-saved weights
Print(__FUNCTION__ + ": ERROR - SaveModelStats failed for " + m_activeFileName + ". Calibration state not persisted.");
SaveShadowNet(currentIndicatorParams);
}
//+------------------------------------------------------------------+
void CExpertSignalAIBase::FinalizeTrainRun(void)
{
//--- deploy the most stable/best-scoring era's weights rather than whatever the run happened to
//--- end on (which may reflect drift after the objective was first hit, or an aborted run). Restore
//--- is now the in-MEMORY snapshot (CNet::RestoreWeights) - see CaptureWeights' note for why the old
//--- file-based restore couldn't work on the CPU-DLL backend.
if(m_haveOosCheckpoint)
{
if(Net.RestoreWeights())
{
dOosForecast = m_bestOosForecast;
RefreshLatestSignal();
//--- NOT during shutdown. RestoreWeights() above is an in-MEMORY swap, so the best checkpoint is
//--- already the live net by this line - and OnDeinit's PersistWeightsOnShutdown() is about to
//--- write exactly those weights anyway. Persisting here too means TWO full ~1MB model writes per
//--- signal on the shutdown path, ahead of the chart cleanup, which is what put OnDeinit over
//--- MetaTrader's budget: measured 4.46 s to "Abnormal termination" on 2026-08-01, with the chart
//--- cleanup completing 0.2 s AFTER the kill. Nothing is lost by skipping it; the same bytes reach
//--- the same file one call later.
if(!m_shutdownInProgress)
PersistDeployedModel();
}
}
//--- Clean up any legacy on-disk checkpoint from an older (file-based) build so it can't linger.
int checkpointFlags = m_activeFileCommon ? FILE_COMMON : 0;
if(FileIsExist(m_activeFileName + "_ckpt.tmp", checkpointFlags))
FileDelete(m_activeFileName + "_ckpt.tmp", checkpointFlags);
//--- (dtStudied used to be held back while scoring a throwaway candidate - that marker belongs
//--- to the DEPLOYED model's "studied up to" state; a candidate eval must leave it untouched. The
//--- checkpoint block above is already inert in eval mode (m_haveOosCheckpoint stays false).
if(m_eraCount > 0)
dtStudied = m_lastBarTime;
m_trainRunActive = false;
m_eraResumePending = false;
m_haveOosCheckpoint = false;
//--- Persist the arrows now drawn on the chart so a deploy/stop survives a later re-add/recompile
//--- without a retrain (durable even if the terminal never gets a clean OnDeinit).
//--- The prune is suppressed when a STOP is in flight, because that means StopTraining() called us and
//--- ShutdownChartCleanup() is about to bulk-purge every arrow anyway. Without this, removing a chart
//--- MID-ERA takes the expensive path twice: once here and once in the cleanup that follows, both
//--- before anything has been cleared. Same defect as the shutdown prune, one call site earlier - see
//--- the prune block in SaveChartSignals() for the measurement.
SaveChartSignals(!m_trainingStopRequested);
}
#endif // WARRIOR_AIBASE_TRAINING_MQH
Reference in a new issue View git blame Copy permalink