marzino/Warrior_EA
Watch
1
0
Fork
You've already forked Warrior_EA
 0
forked from animatedread/Warrior_EA
Code Pull requests Activity
Warrior_EA/Expert/AIBase/Training.mqh

4243 lines
293 KiB
MQL5
Raw Permalink Normal View History

feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//+------------------------------------------------------------------+
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//| Warrior_EA |
//| AnimateDread |
//| |
//| Era loop, plateau ladder, checkpoint selection, deploy/finalise.|
//| |
//| PARTIAL IMPLEMENTATION FILE - not standalone. |
//| This holds CExpertSignalAIBase method BODIES only. The class |
//| declaration lives in Expert\ExpertSignalAIBase.mqh, which |
//| #includes this file at the bottom, after the declaration. Do not |
//| include it anywhere else and do not compile it on its own. |
//| |
//| Split out purely to make the 8216-line original navigable; the |
//| code inside was moved verbatim, not rewritten. |
//+------------------------------------------------------------------+
#ifndef WARRIOR_AIBASE_TRAINING_MQH
#define WARRIOR_AIBASE_TRAINING_MQH
//+------------------------------------------------------------------+
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
//| Upper tail of the standard normal - see the declaration. |
//+------------------------------------------------------------------+
double CExpertSignalAIBase::NormalUpperTail(double z)
{
if(!MathIsValidNumber(z))
return 1.0; // unusable input reads as "not significant"
if(z < 0.0)
return 1.0 - NormalUpperTail(-z);
//--- ntB* / ntP, not the b1..b5 / p the reference prints: AI\Network.mqh line 79 does
//--- "#define b1 AdamBeta1" (and b2 likewise), so a local named b1 here is macro-expanded into the
//--- Adam beta INPUT and the compiler warns that it hides a global. Renamed rather than un-defining
//--- the macro, which the whole Adam path reads.
const double ntP = 0.2316419;
const double ntB1 = 0.319381530, ntB2 = -0.356563782, ntB3 = 1.781477937;
const double ntB4 = -1.821255978, ntB5 = 1.330274429;
double t = 1.0 / (1.0 + ntP * z);
double pdf = MathExp(-0.5 * z * z) / MathSqrt(2.0 * M_PI);
double poly = t * (ntB1 + t * (ntB2 + t * (ntB3 + t * (ntB4 + t * ntB5))));
return MathMax(0.0, MathMin(1.0, pdf * poly));
}
//+------------------------------------------------------------------+
//| Does the checkpoint about to deploy survive having been CHOSEN? |
//| |
//| The per-era test (EDGE_MIN_SIGMAS, see tradeableOK) asks "is this |
//| era's edge more than 2 standard errors above chance". Asked once, |
//| that is a fair question. Asked of every era in a run, and then |
//| answered with the best one, it is the null-of-the-maximum error |
//| this project has now found in four separate places - and this is |
//| the instance that ships a model to a live account. |
//| |
//| Same shape as ReportBarrierGeometryScan's winner test and the |
//| indicator tuner's Sidak correction, applied to the era search: |
//| z = (precision - chance) / SE, SE = sqrt(p0(1-p0)/n) |
//| p_single = P(Z >= z) |
//| p_family = 1 - (1 - p_single)^N |
//| and deployment needs p_family <= DEPLOY_FAMILY_WISE_ALPHA. |
//| |
//| Uses the checkpoint's OWN snapshotted precision/chance/call count, |
//| not the latest era's, because the model that deploys is the one |
//| that has to clear the bar. |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::BestCheckpointSurvivesSelection(double &zObs, double &pFamily, int &nTried)
{
zObs = 0.0;
pFamily = 1.0;
nTried = MathMax(m_deployCandidateEras, 1);
//--- No ranked era yet, or a degenerate chance rate: nothing to test, so nothing to deploy.
if(m_bestDirCalls <= 0 || m_bestDirPrecPct < 0.0 || m_bestChancePrecPct <= 0.0 || m_bestChancePrecPct >= 100.0)
return false;
double p0 = m_bestChancePrecPct / 100.0;
double se = 100.0 * MathSqrt(p0 * (1.0 - p0) / m_bestDirCalls);
if(se <= 0.0)
return false;
zObs = (m_bestDirPrecPct - m_bestChancePrecPct) / se;
double pSingle = NormalUpperTail(zObs);
//--- 1-(1-p)^N directly. At the magnitudes in play (p ~ 1e-4..1e-2, N ~ 10..1000) double precision is
//--- ample; no need for the log1p/expm1 form MQL5 would not give us anyway.
pFamily = 1.0 - MathPow(1.0 - pSingle, (double)nTried);
return (pFamily <= DEPLOY_FAMILY_WISE_ALPHA);
}
//+------------------------------------------------------------------+
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//| ENSEMBLE GATE - the same test as above, asked of the VOTE. |
//| See the ENSEMBLE DEPLOY GATE block in ExpertSignalAIBase.mqh for |
//| why the vote rather than the member is the thing being gated. |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::EnsembleSurvivesSelection(double &zObs, double &pFamily, int &nTried)
{
zObs = 0.0;
pFamily = 1.0;
nTried = MathMax(g_ensCandidateEras, 1);
if(g_ensBestCalls <= 0 || g_ensBestPrecPct < 0.0 || g_ensBestChancePct <= 0.0 || g_ensBestChancePct >= 100.0)
return false;
double p0 = g_ensBestChancePct / 100.0;
double se = 100.0 * MathSqrt(p0 * (1.0 - p0) / g_ensBestCalls);
if(se <= 0.0)
return false;
zObs = (g_ensBestPrecPct - g_ensBestChancePct) / se;
pFamily = 1.0 - MathPow(1.0 - NormalUpperTail(zObs), (double)nTried);
return (pFamily <= DEPLOY_FAMILY_WISE_ALPHA);
}
//+------------------------------------------------------------------+
//| JOINT CHECKPOINT: snapshot EVERY member's weights, at this one |
//| era, and commit each member's own era statistics as the stats its |
//| best checkpoint is described by. |
//| |
//| Correct only because of the era barrier - see the ENSEMBLE DEPLOY |
//| GATE block. Every member has finished era `votedEra` and is held |
//| before era votedEra+1, so all four sets of weights belong to the |
//| same measured instant. Without the barrier this would snapshot |
//| whatever era each member happened to be mid-way through. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::EnsembleCommitJointCheckpoint(const long votedEra)
{
int captured = 0, members = 0;
for(int i = 0; i < ArraySize(g_warriorEnsemble); i++)
{
CExpertSignalAIBase *mm = g_warriorEnsemble[i];
if(CheckPointer(mm) == POINTER_INVALID || mm.m_ensembleIndex < 0)
continue;
if(mm.m_trainingComplete || mm.m_trainingStopRequested || mm.m_trainingPaused || !mm.m_isInitialized)
continue;
members++;
//--- The member's OWN figures at the winning era. They describe this member's contribution to a
//--- checkpoint the ENSEMBLE selected, which is why they are committed from the stash rather
//--- than from a per-member ranking: no member "won" this era, the vote did.
mm.m_bestOosForecast = mm.m_eraStatBlended;
mm.m_bestBalancedOos = mm.m_eraStatScore;
mm.m_bestPassedRecall = mm.m_eraStatTradeable;
mm.m_bestBothSidesLive = mm.m_eraStatTwoSided;
mm.m_bestDirPrecPct = mm.m_eraStatPrecPct;
mm.m_bestChancePrecPct = mm.m_eraStatChancePct;
mm.m_bestDirCalls = mm.m_eraStatCalls;
mm.m_bestDirConfThreshold = mm.m_eraStatThreshold;
//--- In-memory snapshot, same primitive the solo path uses. A member whose capture fails keeps
//--- m_haveOosCheckpoint false and is reported - it would otherwise deploy whatever weights it
//--- happens to hold at the end of the run, silently breaking the "deploy what was measured"
//--- guarantee this whole mechanism exists for.
if(CheckPointer(mm.Net) != POINTER_INVALID && mm.Net.CaptureWeights())
{
mm.m_haveOosCheckpoint = true;
mm.m_checkpointEra = votedEra; // the deploy gate cross-checks this against the winning era
captured++;
}
else
Print(mm.ID + ": WARNING - joint ensemble checkpoint capture FAILED at era " +
IntegerToString((int)votedEra) + ". This member cannot contribute the weights the vote"
" was measured with; the ensemble will not deploy a checkpoint it cannot reproduce.");
//--- a new joint best retires the shared ladder for everyone
mm.m_erasSinceBestBalanced = 0;
mm.m_plateauStage = 0;
mm.m_restartBoostErasLeft = 0;
mm.m_consecutiveRegressions = 0;
}
//--- PARTIAL CAPTURE IS NOT A CHECKPOINT. Retire this era as the best rather than leaving the
//--- record pointing at a quartet that cannot be reproduced - the era-stamp test would refuse to
//--- deploy it anyway, and leaving a high score in place would block every later era from winning,
//--- freezing the search behind a checkpoint that does not exist. Rolling it back lets the run
//--- carry on and simply find its best again.
if(captured < members)
{
g_ensBestScore = -1.0;
g_ensBestTradeable = false;
g_ensBestTwoSided = false;
g_ensBestCalls = 0;
g_ensBestEra = -1;
Print("AI ensemble: joint checkpoint INCOMPLETE at era " + IntegerToString((int)votedEra) +
" (" + IntegerToString(captured) + " of " + IntegerToString(members) + " members captured)"
" - discarding this era as the best; the search continues from no joint checkpoint.");
}
}
//+------------------------------------------------------------------+
//| Once per era, on the LAST still-training member to finish its |
//| pass-3 scan: score the combined vote, rank the era, checkpoint, |
//| advance the shared plateau ladder, and decide deployment. |
//| |
//| Every statistic mirrors the per-member gate exactly (coverage |
//| floor, chance reference, EDGE_MIN_SIGMAS margin, Sidak |
//| correction); only the population differs - the bars the VOTE |
//| fired on rather than the bars one member called. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::EnsembleEraVerdict(const int needMask, const long votedEra, double &etaLocal)
{
fix(gate): the ensemble gate certified a vote the EA never casts g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on the same criterion the live trade does". It did not. Two independent mismatches, both silent: CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100 number straight off the softmax head. Live contributes m_weight x the tier's pattern weight, and BOTH of those are rewritten from the signal DB by UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share an axis and nothing relates them, so the same bar was one number to the gate and a different one to the order path. Same shape as the 2026-08-09 geometry incident: certified on one game, paid on another. DENOMINATOR. The gate divided by the member count, so an abstaining member pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero contribution in BOTH the sum and the count (`if(direction == 0) continue;` before `number++`) - live is a mean over VOTERS. The gate was therefore scoring a strictly more agreement-heavy set of bars than the EA trades. The contribution hook's own comment asserted the opposite ("abstentions dilute the average exactly as they do in the live vote"), while the AI_CHOICE enum 20 lines away correctly documented union semantics. LiveVoteContribution() is now the single definition of "what this member votes", called from the gate; the live path reaches the same arithmetic through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually voted, separately from who evaluated the bar, because those are the divisor and the shared-population test respectively. ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar wrapper - the OOS scan holds the scanned bar's decision in a local, and dPrevSignal is a different bar. NOT changed, deliberately: the gate still does not model live NMS declustering, and the per-member solo gate still scores every directional call rather than threshold-clearing ones. Both are selection-metric changes and this codebase has twice been bitten by switching one blind. Also corrects a stale paragraph in m_pattern_0's declaration block quoting 80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has been since the confidence floor and alternation gate were removed; the block carried both tables at once, and the dead one was quoted back as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
int members = EnsembleBitCount(needMask);
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//--- One trainer left (the others deployed, paused or stopped) is not an ensemble read: the
//--- "vote" would be that member's own signal and the gate would silently become the solo gate
//--- under an ensemble label. Members keep training; nothing is ranked or deployed from here.
if(members < 2)
return;
//--- SHARED BARS ONLY. A bar one member skipped (feature-window failure) has an average over a
//--- different membership, which is a different quantity - averaging it in would make the score
//--- depend on which member happened to fail where.
int shared = 0, fired = 0, wins = 0, firedLong = 0, firedShort = 0;
feat(ensemble): per-NN inputs replace the preset selector - the meta head becomes the vote's gate User design (2026-08-19): 'remove the enum menu that selects neural networks... individual inputs for every NN just like classic signals... the META NN should be integrated into the voting decision pipeline when enabled... as a bonus meta labelling is applied to enabled NNs.' - AI_CHOICE is GONE (tombstoned per the stale-.set doctrine). Use_MLP/Use_CONV/Use_LSTM/ Use_CONVLSTM are ordinary bools like the classic votes; the ensemble arithmetic adapts to any subset because the consensus divisor is the enabled capable weight. Two or more enabled = ensemble (|ENS1 token + joint gate, exactly the old AI_HYBRID fingerprints, so existing weight files keep loading); one = the old solo preset; none = classic-only. - Use_MetaLabeling un-couples META from the direction NNs (the old selector made them mutually exclusive). S3 ships: CSignalMETA::LiveMetaGate scores each vote-cleared entry (shared window at bar 1 + proposal descriptor: side, net vote, live geometry, spread/ATR; pattern one-hot ZEROED - ranking, not calibrated probability, documented in the body) and vetoes below the cost-adjusted break-even. Entries only; fail-open everywhere, loudly. - COEXISTENCE HAZARDS closed: VoteCapableWeight()=0 and ProspectiveVote()=false for the meta target - solo-only until today, a trained META would otherwise sit in the consensus divisor as a permanent abstainer and shrink every vote by its module weight. - CERTIFIED == TRADED: the ensemble era verdict replays the identical veto through the same g_warriorMetaGate pointer over its OOS fired bars (bar re-resolved from the row's own time; fail-open counted as fires and reported: 'metaGate: N approved, M vetoed, K unscored'). The overlay deliberately does NOT replay it (veto-filter-in-replay class, calendar-cliff precedent) - documented at the sweep site. Solo charts' own gate does not model the veto - the standing solo-gate caveat, documented at the input. - DB continuity: the pattern/journal DB fingerprint's first slot was (int)AIType; DbLegacyAiSlot() maps every legacy-expressible config to its OLD value (new 2-3 member subsets get 100+bitmask, outside the legacy range) so no existing database re-keys. filterID becomes the enabled roster via one EnabledNNSummary(). - HUD: the meta line shows the gate (armed/(trn), last P vs BE, ok/veto tally); the armed/disarmed announcement fires on state change via one latch (MetaGateArmedNow), not only when an entry happens to be proposed. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 13:01:02 -04:00
int metaOk = 0, metaVetoed = 0, metaOpen = 0;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
int dirLabelBars = 0, alwaysLongWins = 0, alwaysShortWins = 0;
for(int r = 0; r < g_ensVoteRows; r++)
{
if((g_ensVoteMask[r] & needMask) != needMask)
continue;
shared++;
if(g_ensVoteDirLabel[r])
dirLabelBars++;
//--- zero-skill reference, measured over EVERY shared bar (see chancePrecPct's derivation in
//--- the era-end block): what always-long and always-short would have collected here
if(g_ensVoteWinLong[r])
alwaysLongWins++;
if(g_ensVoteWinShort[r])
alwaysShortWins++;
fix(gate): the ensemble gate certified a vote the EA never casts g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on the same criterion the live trade does". It did not. Two independent mismatches, both silent: CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100 number straight off the softmax head. Live contributes m_weight x the tier's pattern weight, and BOTH of those are rewritten from the signal DB by UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share an axis and nothing relates them, so the same bar was one number to the gate and a different one to the order path. Same shape as the 2026-08-09 geometry incident: certified on one game, paid on another. DENOMINATOR. The gate divided by the member count, so an abstaining member pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero contribution in BOTH the sum and the count (`if(direction == 0) continue;` before `number++`) - live is a mean over VOTERS. The gate was therefore scoring a strictly more agreement-heavy set of bars than the EA trades. The contribution hook's own comment asserted the opposite ("abstentions dilute the average exactly as they do in the live vote"), while the AI_CHOICE enum 20 lines away correctly documented union semantics. LiveVoteContribution() is now the single definition of "what this member votes", called from the gate; the live path reaches the same arithmetic through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually voted, separately from who evaluated the bar, because those are the divisor and the shared-population test respectively. ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar wrapper - the OOS scan holds the scanned bar's decision in a local, and dPrevSignal is a different bar. NOT changed, deliberately: the gate still does not model live NMS declustering, and the per-member solo gate still scores every directional call rather than threshold-clearing ones. Both are selection-metric changes and this codebase has twice been bitten by switching one blind. Also corrects a stale paragraph in m_pattern_0's declaration block quoting 80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has been since the confidence floor and alternation gate were removed; the block carried both tables at once, and the dead one was quoted back as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
//--- THE LIVE AGGREGATION, reproduced exactly (CExpertSignalCustom::Direction(), pass 2 plus the
//--- `result /= number` normalization): sum the members' signed votes, divide by how many of them
//--- ACTUALLY VOTED, and compare the magnitude against Min_Vote_Open on the same 0..100 scale the
//--- tier weights already live on. No x100 any more - the contributions are pattern weights now,
//--- not confidences (see the oEnsembleVote comment in the pass-3 scan).
//---
//--- Dividing by `members` was the old behaviour and it is NOT what live does: an abstaining
//--- member was pulling the average toward zero here while live simply left it out, so the gate
//--- fired on a strictly smaller, more agreement-heavy set of bars than the EA trades.
//---
feat(vote): CONSENSUS arithmetic - agreement is now what the threshold dials Era-680 report, all three observations one equation: "peak 29, no arrows at threshold 30" / "at 20, arrows on EVERY bar" / "label at 12 while arrows everywhere". Under the voters-only divisor, any bar with at least one directional voter read the weighted mean of the firing tiers' weights - and once the tiers self-ranked to each model's pooled win rate (~28-31), that mean was NEAR-CONSTANT regardless of headcount. One member alone: ~29. Four unanimous: ~29. Min_Vote_Open was a step function around that constant - above it nothing ever fired, below it everything did - and the label's 12 was a 3v1 split netting through the same divisor. Not three display bugs: one arithmetic that could not express agreement. The divisor is now the CAPABLE weight - every filter that could vote, whether it did or not: * live (Direction): VoteCapableWeight() - classic pattern ladders always, veto filters never, AI members once past the same readiness test LongCondition gates on. A model still training must not dilute an ensemble it cannot join: four trainees + one deployed model is a solo chart wearing an ensemble label, and the solo vote reads full strength. * gate (EnsembleEraVerdict): g_ensVoteWeightSum accumulates for every member that EVALUATED the bar, Neutral included. * overlay sweep + prospective readout: weight counts whenever the member has data; a snapshotted Neutral dilutes. One arithmetic, four sites, same numbers everywhere. What the numbers become (four members, w~0.29, tiers~29): unanimous ~29 - the CEILING, which is the pooled win rate and is what the peak displays; 3-of-4 ~22; 2-of-4 ~14.5; 3v1 ~14.5. Min_Vote_Open 20 now means "roughly three-quarters of the ensemble's trust agrees, net". It MUST sit below the ceiling to ever fire - the census/peak states the ceiling. This is the ensemble the user specified in the original design discussion ("if the perceptron also votes, both together reach the threshold; if another NN votes the other side, the threshold is not reached") - union semantics was the pre-ensemble behaviour, kept until measurement showed its vote magnitude was a constant. Plus overlay DECLUSTERING, the other half of "arrows on every bar": the same three NMS rules as the per-member arrows (same-direction runs collapse to their first bar, cross-direction flicker keeps the stronger side), online over the sweep's strictly oldest->newest walk. Suppression is a verdict and deletes a standing arrow; the den==0 no-data skip still never does. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 06:58:03 -04:00
//--- CONSENSUS (2026-08-19): the divisor is the weight of every member that EVALUATED the bar,
//--- Neutral included, so agreement is what the magnitude measures - full agreement reads the
//--- members' weighted mean win rate (the ceiling), one-of-four reads a quarter of it, splits
//--- net out. Union semantics (voters-only divisor) made the magnitude near-constant at the
//--- pooled win rate once tiers self-ranked, and the threshold a step function around it.
fix(gate): the ensemble gate certified a vote the EA never casts g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on the same criterion the live trade does". It did not. Two independent mismatches, both silent: CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100 number straight off the softmax head. Live contributes m_weight x the tier's pattern weight, and BOTH of those are rewritten from the signal DB by UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share an axis and nothing relates them, so the same bar was one number to the gate and a different one to the order path. Same shape as the 2026-08-09 geometry incident: certified on one game, paid on another. DENOMINATOR. The gate divided by the member count, so an abstaining member pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero contribution in BOTH the sum and the count (`if(direction == 0) continue;` before `number++`) - live is a mean over VOTERS. The gate was therefore scoring a strictly more agreement-heavy set of bars than the EA trades. The contribution hook's own comment asserted the opposite ("abstentions dilute the average exactly as they do in the live vote"), while the AI_CHOICE enum 20 lines away correctly documented union semantics. LiveVoteContribution() is now the single definition of "what this member votes", called from the gate; the live path reaches the same arithmetic through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually voted, separately from who evaluated the bar, because those are the divisor and the shared-population test respectively. ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar wrapper - the OOS scan holds the scanned bar's decision in a local, and dPrevSignal is a different bar. NOT changed, deliberately: the gate still does not model live NMS declustering, and the per-member solo gate still scores every directional call rather than threshold-clearing ones. Both are selection-metric changes and this codebase has twice been bitten by switching one blind. Also corrects a stale paragraph in m_pattern_0's declaration block quoting 80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has been since the confidence floor and alternation gate were removed; the block carried both tables at once, and the dead one was quoted back as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
int voters = EnsembleBitCount(g_ensVoteVoterMask[r] & needMask);
feat(vote): thresholds become confidence percentages, on ONE scale everywhere User request: "the entry/exit thresholds are manual numbers, I would like them to be confidence percentages, so the current 20 would be only 20% confidence in a profitable trade." WHY 20 WAS EVER SENSIBLE. Under UseDatabaseRanking both factors of a filter's contribution are win rates: the pattern weight is that pattern's measured win rate (UpdateSignalsWeights -> ApplyPatternWeight) and m_weight is the filter's average win rate over its patterns, /100. Dividing the sum by the VOTER COUNT therefore produced a mean of PRODUCTS of two win rates - a genuinely 60%-accurate filter firing a 60% pattern scored 0.60 x 60 = 36. The number was never on a probability scale, so its magnitude meant nothing on its own. Dividing by Sum(m_weight) instead makes it a weighted MEAN of win rates, which is a win rate: result = Sum(w_i*p_i)/Sum(w_i). Every voter at 60% now reads 60; MACD's double-divergence pattern (weight 100) voting alone reads 100. m_weight stops being a discount on the probability and becomes how much a filter's opinion COUNTS - which is what a module weight should always have been. Default Min_Vote_Open 20 -> 50: not a tightening, the same bar re-expressed. ONE SCALE, EVERYWHERE - the part that made this bigger than a rescale. Three other places compared against a 0..1 softmax confidence and would each have become a fresh currency mismatch the moment the input changed meaning: * the AI early-exit route (LiveSignedConfidence vs m_ai_exit_threshold) now reads m_lastAiVote - the AI filters' own weighted mean, undiluted by the classic side, which is the only reason that route exists - against the same m_threshold_close the averaged vote uses. m_ai_exit_threshold is retired rather than left dangling. * m_oosDecisionSeries now carries the vote, not the confidence, so the exit SIMULATION stops modelling a close rule the EA does not run. * ExitPolicy() clamped anything > 1.0 to zero. Passing the unscaled input through that would have silently switched vote exits off in the simulation while live went on running them - found before it shipped; the bound now tracks the scale. LiveSignedConfidence() is deliberately untouched and still 0..1: MM sizing, SL/TP scaling and the intelligent trailing want a model confidence, not a win rate. CALIBRATION CAVEAT, stated in the code where the claim is made: this is only a real probability to the extent the pattern weights are. A pattern with fewer than MIN_TRADES_FOR_WIN_RATE journaled trades keeps its DEFAULT weight - a designed prior (25/50/75/100 for the AI tiers), not a measurement. Until the signal DB fills, "60" means "the designed conviction of the patterns that fired". Closing that gap is the next commit. Also corrects VOTE_CLOSE_PRESETS' comment, which documented the two scales this removes. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:52:08 -04:00
if(voters <= 0 || g_ensVoteWeightSum[r] <= 0.0)
fix(gate): the ensemble gate certified a vote the EA never casts g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on the same criterion the live trade does". It did not. Two independent mismatches, both silent: CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100 number straight off the softmax head. Live contributes m_weight x the tier's pattern weight, and BOTH of those are rewritten from the signal DB by UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share an axis and nothing relates them, so the same bar was one number to the gate and a different one to the order path. Same shape as the 2026-08-09 geometry incident: certified on one game, paid on another. DENOMINATOR. The gate divided by the member count, so an abstaining member pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero contribution in BOTH the sum and the count (`if(direction == 0) continue;` before `number++`) - live is a mean over VOTERS. The gate was therefore scoring a strictly more agreement-heavy set of bars than the EA trades. The contribution hook's own comment asserted the opposite ("abstentions dilute the average exactly as they do in the live vote"), while the AI_CHOICE enum 20 lines away correctly documented union semantics. LiveVoteContribution() is now the single definition of "what this member votes", called from the gate; the live path reaches the same arithmetic through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually voted, separately from who evaluated the bar, because those are the divisor and the shared-population test respectively. ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar wrapper - the OOS scan holds the scanned bar's decision in a local, and dPrevSignal is a different bar. NOT changed, deliberately: the gate still does not model live NMS declustering, and the per-member solo gate still scores every directional call rather than threshold-clearing ones. Both are selection-metric changes and this codebase has twice been bitten by switching one blind. Also corrects a stale paragraph in m_pattern_0's declaration block quoting 80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has been since the confidence floor and alternation gate were removed; the block carried both tables at once, and the dead one was quoted back as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
continue; // every member abstained: no vote, no trade, not a fired bar
feat(vote): thresholds become confidence percentages, on ONE scale everywhere User request: "the entry/exit thresholds are manual numbers, I would like them to be confidence percentages, so the current 20 would be only 20% confidence in a profitable trade." WHY 20 WAS EVER SENSIBLE. Under UseDatabaseRanking both factors of a filter's contribution are win rates: the pattern weight is that pattern's measured win rate (UpdateSignalsWeights -> ApplyPatternWeight) and m_weight is the filter's average win rate over its patterns, /100. Dividing the sum by the VOTER COUNT therefore produced a mean of PRODUCTS of two win rates - a genuinely 60%-accurate filter firing a 60% pattern scored 0.60 x 60 = 36. The number was never on a probability scale, so its magnitude meant nothing on its own. Dividing by Sum(m_weight) instead makes it a weighted MEAN of win rates, which is a win rate: result = Sum(w_i*p_i)/Sum(w_i). Every voter at 60% now reads 60; MACD's double-divergence pattern (weight 100) voting alone reads 100. m_weight stops being a discount on the probability and becomes how much a filter's opinion COUNTS - which is what a module weight should always have been. Default Min_Vote_Open 20 -> 50: not a tightening, the same bar re-expressed. ONE SCALE, EVERYWHERE - the part that made this bigger than a rescale. Three other places compared against a 0..1 softmax confidence and would each have become a fresh currency mismatch the moment the input changed meaning: * the AI early-exit route (LiveSignedConfidence vs m_ai_exit_threshold) now reads m_lastAiVote - the AI filters' own weighted mean, undiluted by the classic side, which is the only reason that route exists - against the same m_threshold_close the averaged vote uses. m_ai_exit_threshold is retired rather than left dangling. * m_oosDecisionSeries now carries the vote, not the confidence, so the exit SIMULATION stops modelling a close rule the EA does not run. * ExitPolicy() clamped anything > 1.0 to zero. Passing the unscaled input through that would have silently switched vote exits off in the simulation while live went on running them - found before it shipped; the bound now tracks the scale. LiveSignedConfidence() is deliberately untouched and still 0..1: MM sizing, SL/TP scaling and the intelligent trailing want a model confidence, not a win rate. CALIBRATION CAVEAT, stated in the code where the claim is made: this is only a real probability to the extent the pattern weights are. A pattern with fewer than MIN_TRADES_FOR_WIN_RATE journaled trades keeps its DEFAULT weight - a designed prior (25/50/75/100 for the AI tiers), not a measurement. Until the signal DB fills, "60" means "the designed conviction of the patterns that fired". Closing that gap is the next commit. Also corrects VOTE_CLOSE_PRESETS' comment, which documented the two scales this removes. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:52:08 -04:00
double net = g_ensVoteSum[r] / g_ensVoteWeightSum[r];
fix(gate): the ensemble gate certified a vote the EA never casts g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on the same criterion the live trade does". It did not. Two independent mismatches, both silent: CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100 number straight off the softmax head. Live contributes m_weight x the tier's pattern weight, and BOTH of those are rewritten from the signal DB by UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share an axis and nothing relates them, so the same bar was one number to the gate and a different one to the order path. Same shape as the 2026-08-09 geometry incident: certified on one game, paid on another. DENOMINATOR. The gate divided by the member count, so an abstaining member pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero contribution in BOTH the sum and the count (`if(direction == 0) continue;` before `number++`) - live is a mean over VOTERS. The gate was therefore scoring a strictly more agreement-heavy set of bars than the EA trades. The contribution hook's own comment asserted the opposite ("abstentions dilute the average exactly as they do in the live vote"), while the AI_CHOICE enum 20 lines away correctly documented union semantics. LiveVoteContribution() is now the single definition of "what this member votes", called from the gate; the live path reaches the same arithmetic through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually voted, separately from who evaluated the bar, because those are the divisor and the shared-population test respectively. ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar wrapper - the OOS scan holds the scanned bar's decision in a local, and dPrevSignal is a different bar. NOT changed, deliberately: the gate still does not model live NMS declustering, and the per-member solo gate still scores every directional call rather than threshold-clearing ones. Both are selection-metric changes and this codebase has twice been bitten by switching one blind. Also corrects a stale paragraph in m_pattern_0's declaration block quoting 80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has been since the confidence floor and alternation gate were removed; the block carried both tables at once, and the dead one was quoted back as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
if(MathAbs(net) < g_ensembleVoteThreshold)
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
continue;
feat(consistency): the five review flaws fixed - training wears the live constraints, the gate wears the policy 1. Labels and the exit simulator go through the broker's stop-distance check: risk/reward widen to SYMBOL_TRADE_STOPS_LEVEL exactly as TCAdjustStops does at order time - the M5/tight-ATR case where live trades ran wider geometry than training measured. Current stops level stands in for history (like the spread); measured quantity, so it does not key the fingerprint. 2. The Intelligent drift verdict moved into RefreshDriftVerdict(), which RESCANS the label cache and now runs at every era end beside RankTiersFromOos - era-cadence instead of waiting for rare full rebuilds. Prints only on change. 3. Session filter is any-broker: sessions defined on their financial centres' civil clocks (London 08-16 Europe/London, NY 08-17 America/New_York, Tokyo 09-18 Asia/Tokyo), converted to UTC by each centre's own computed DST rule (EU last-Sun-Mar/Oct, US 2nd-Sun-Mar/1st-Sun-Nov), then to broker time by the MEASURED server-vs-GMT offset (half-hour brokers included). Windows may wrap midnight in broker time - the interval test handles it. Replaces the EET-hardcoded anchors, which were correct on exactly one broker and got Tokyo wrong by an hour each European summer. 4. The current-session-table-for-history caveat resolved by analysis: the bars bound the error - a too-late assumed close meets no bars (zero error), a too-early one truncates conservatively (<=1h, never optimistic, cannot manufacture edge). Documented at the site. 5. The ensemble deploy gate mirrors the direction policy: blocked-side fires are not fired bars (certified == traded), the zero-skill reference uses only ACHIEVABLE baselines (always-short is not a strategy a long-only book can run), and one-sidedness BY POLICY is not degeneracy - the two-sided requirement applies only when both sides are allowed. Sell predictions keep their other jobs (exit triggers, consensus dilution) untouched. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 12:14:01 -04:00
//--- THE DIRECTION POLICY IS PART OF WHAT GETS CERTIFIED (2026-08-19). Under LONG_ONLY/
//--- SHORT_ONLY or an Intelligent drift verdict, live never places the blocked side's
//--- trades - scoring them here would certify a vote the EA does not cast, the exact
//--- certified!=traded defect this gate was rebuilt to end (2c443ba). Sell PREDICTIONS
//--- keep their other jobs untouched (exit trigger for open longs, consensus dilution);
//--- only their ENTRY fires stop counting, mirroring CheckOpenLong/Short exactly.
if(!WarriorDirectionAllows(net > 0.0))
continue;
feat(ensemble): per-NN inputs replace the preset selector - the meta head becomes the vote's gate User design (2026-08-19): 'remove the enum menu that selects neural networks... individual inputs for every NN just like classic signals... the META NN should be integrated into the voting decision pipeline when enabled... as a bonus meta labelling is applied to enabled NNs.' - AI_CHOICE is GONE (tombstoned per the stale-.set doctrine). Use_MLP/Use_CONV/Use_LSTM/ Use_CONVLSTM are ordinary bools like the classic votes; the ensemble arithmetic adapts to any subset because the consensus divisor is the enabled capable weight. Two or more enabled = ensemble (|ENS1 token + joint gate, exactly the old AI_HYBRID fingerprints, so existing weight files keep loading); one = the old solo preset; none = classic-only. - Use_MetaLabeling un-couples META from the direction NNs (the old selector made them mutually exclusive). S3 ships: CSignalMETA::LiveMetaGate scores each vote-cleared entry (shared window at bar 1 + proposal descriptor: side, net vote, live geometry, spread/ATR; pattern one-hot ZEROED - ranking, not calibrated probability, documented in the body) and vetoes below the cost-adjusted break-even. Entries only; fail-open everywhere, loudly. - COEXISTENCE HAZARDS closed: VoteCapableWeight()=0 and ProspectiveVote()=false for the meta target - solo-only until today, a trained META would otherwise sit in the consensus divisor as a permanent abstainer and shrink every vote by its module weight. - CERTIFIED == TRADED: the ensemble era verdict replays the identical veto through the same g_warriorMetaGate pointer over its OOS fired bars (bar re-resolved from the row's own time; fail-open counted as fires and reported: 'metaGate: N approved, M vetoed, K unscored'). The overlay deliberately does NOT replay it (veto-filter-in-replay class, calendar-cliff precedent) - documented at the sweep site. Solo charts' own gate does not model the veto - the standing solo-gate caveat, documented at the input. - DB continuity: the pattern/journal DB fingerprint's first slot was (int)AIType; DbLegacyAiSlot() maps every legacy-expressible config to its OLD value (new 2-3 member subsets get 100+bitmask, outside the legacy range) so no existing database re-keys. filterID becomes the enabled roster via one EnabledNNSummary(). - HUD: the meta line shows the gate (armed/(trn), last P vs BE, ok/veto tally); the armed/disarmed announcement fires on state change via one latch (MetaGateArmedNow), not only when an entry happens to be proposed. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 13:01:02 -04:00
//--- THE META GATE IS PART OF WHAT GETS CERTIFIED (2026-08-19), same doctrine as the
//--- direction policy above: live, every vote-cleared entry passes LiveMetaGate before it
//--- can trade, so the verdict replays the identical veto through the identical pointer or
//--- it certifies fires the EA declines. The bar index is re-resolved from the row's own
//--- bar-open time (exact match required): indices shift with every closed bar, times do
//--- not. Fail-open codes COUNT AS FIRES - unscorable this deep or gate not armed, live
//--- they would trade - and are tallied separately so the era line says how much of the
//--- certified set the gate actually scored.
if(g_warriorMetaGate != NULL)
{
double mgP = -1.0, mgBe = -1.0;
int mgBar = iBarShift(m_symbol.Name(), (ENUM_TIMEFRAMES)m_period, g_ensVoteTime[r], true);
//--- mgBar > 1, not > 0: barIdx 1 is LiveMetaGate's "live entry" telemetry key, so a
//--- replay that resolves to the newest closed bar is left unscored rather than allowed to
//--- masquerade as a live approval/veto in the HUD counters.
int mgV = (mgBar > 1) ? g_warriorMetaGate.LiveMetaGate(net > 0.0, net, mgP, mgBe, mgBar) : 1;
if(mgV < 0)
{
metaVetoed++;
continue;
}
if(mgV == 2)
metaOk++;
else
metaOpen++;
}
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
fired++;
if(net > 0.0)
{
firedLong++;
if(g_ensVoteWinLong[r])
wins++;
}
else
{
firedShort++;
if(g_ensVoteWinShort[r])
wins++;
}
}
double slBe = 0.0, tpBe = 0.0;
BarrierMultiples(slBe, tpBe);
int bePct = (slBe > 0.0 && tpBe > 0.0) ? (int)MathRound(100.0 * slBe / (slBe + tpBe)) : -1;
bool measurable = (shared > 0 && dirLabelBars > 0);
double votePrecPct = (fired > 0) ? 100.0 * wins / fired : -1.0;
double coveragePct = measurable ? 100.0 * fired / shared : -1.0;
double baseRatePct = measurable ? 100.0 * dirLabelBars / shared : -1.0;
double minCoverPct = measurable ? baseRatePct * MIN_COVERAGE_FRACTION_OF_BASE_RATE : -1.0;
feat(consistency): the five review flaws fixed - training wears the live constraints, the gate wears the policy 1. Labels and the exit simulator go through the broker's stop-distance check: risk/reward widen to SYMBOL_TRADE_STOPS_LEVEL exactly as TCAdjustStops does at order time - the M5/tight-ATR case where live trades ran wider geometry than training measured. Current stops level stands in for history (like the spread); measured quantity, so it does not key the fingerprint. 2. The Intelligent drift verdict moved into RefreshDriftVerdict(), which RESCANS the label cache and now runs at every era end beside RankTiersFromOos - era-cadence instead of waiting for rare full rebuilds. Prints only on change. 3. Session filter is any-broker: sessions defined on their financial centres' civil clocks (London 08-16 Europe/London, NY 08-17 America/New_York, Tokyo 09-18 Asia/Tokyo), converted to UTC by each centre's own computed DST rule (EU last-Sun-Mar/Oct, US 2nd-Sun-Mar/1st-Sun-Nov), then to broker time by the MEASURED server-vs-GMT offset (half-hour brokers included). Windows may wrap midnight in broker time - the interval test handles it. Replaces the EET-hardcoded anchors, which were correct on exactly one broker and got Tokyo wrong by an hour each European summer. 4. The current-session-table-for-history caveat resolved by analysis: the bars bound the error - a too-late assumed close meets no bars (zero error), a too-early one truncates conservatively (<=1h, never optimistic, cannot manufacture edge). Documented at the site. 5. The ensemble deploy gate mirrors the direction policy: blocked-side fires are not fired bars (certified == traded), the zero-skill reference uses only ACHIEVABLE baselines (always-short is not a strategy a long-only book can run), and one-sidedness BY POLICY is not degeneracy - the two-sided requirement applies only when both sides are allowed. Sell predictions keep their other jobs (exit triggers, consensus dilution) untouched. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 12:14:01 -04:00
//--- The zero-skill reference must be ACHIEVABLE under the direction policy: with shorts
//--- blocked, always-short is not a strategy anyone could run, and ranking the vote against it
//--- would score a long-only book against a baseline the policy forbids. Both sides allowed =
//--- the larger baseline, exactly as before.
double chancePct = -1.0;
if(measurable)
{
double chanceL = 100.0 * alwaysLongWins / shared;
double chanceS = 100.0 * alwaysShortWins / shared;
bool allowL = WarriorDirectionAllows(true);
bool allowS = WarriorDirectionAllows(false);
chancePct = (allowL && allowS) ? MathMax(chanceL, chanceS)
: (allowL ? chanceL : (allowS ? chanceS : MathMax(chanceL, chanceS)));
}
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
double chanceP = (chancePct >= 0.0) ? chancePct / 100.0 : 0.0;
fix(labels): overlapping-label sample correction + horizon cap on the scale ladder Three defects, all surfaced by the 2026-08-17 SP500 H4 run that shipped stop 4.86 / target 9.71 (width 14.57*ATR, horizon 384). 1. EVERY STANDARD ERROR ASSUMED INDEPENDENT SAMPLES. Triple-barrier labels started one per bar overlap by the label's lifespan, so n calls are worth ~n/L independent observations (Lopez de Prado, AFML ch. 4 - sample uniqueness). All three sqrt(p(1-p)/n) sites divided by the RAW count. The tell: the operating point's null-of-the-maximum gate is family-wise and should fire on ~5% of eras under the null. Measured fire rates - PAI 47/73 (64%), ConvLSTM 9/24, LSTM 8/21 (38%), CONV 4/62 (6%). CONV, the only model whose margin distribution admits few bins, sat on the null; the rest cleared a bar that was too low by ~sqrt(L). PAI's deployed threshold consequently alternated between the ENDS of its own range era to era (0.10 -> 0.88 -> 0.86 -> 0.66; coverage 16% <-> 73%). TripleBarrierLabel now records when each label became KNOWABLE - the first winning touch, or both stops, or the timeout - and the prebuild accumulates the mean. EffectiveSampleSize() feeds the operating point, the member deploy gate and the ensemble vote gate. Conservative by construction (n/L is an upper bound on the damage); gates get harder, never easier. 2. THE SCALE LADDER RAN AWAY, again. Horizon scales as swingMedian*sl*tp, and since 4d8cb08 reachability is measured OVER that horizon - so a wider rung buys itself the time that makes it look reachable. Same target -> horizon -> reach -> target loop the excursion window is kept short to avoid; fixing the window confusion reopened it through the other door. It walked 128 -> 256 -> 384 bars and stopped at q90, the widest rung there is, with every rung reading 39-48% against a 20% floor. A floor nothing fails selects nothing. Rungs whose required horizon exceeds BARRIER_HORIZON_MAX are now rejected - the same rule ReportGeometryExpectancyScan already applied. It was printing the shipped pair as CLAMPED and disqualified ('h384!') two lines under the deriver that chose it: two subsystems, one geometry, opposite verdicts. 3. THE RUNG SNAP DESTROYED THE RATIO IT WAS COMPARING. Both legs snapped independently to the coarse first-passage grid, re-rating each candidate: q90 4.86/9.71 -> 5.00/10.00 (2.00), q85 4.07/8.14 -> 5.00/10.00 (IDENTICAL measurement), q75 3.07/6.13 -> 4.00/6.50 (1.63 - a nearer target). So the ladder compared win shares taken at ratios from 1.63 to 2.17 and read the differences as scale. It is why the reach column came out non-monotone in width (q75 48.5% above q90 42.9%). The stop now snaps to its nearest rung in log space and the target follows the ratio off it; the pair actually measured is returned and logged, so a collision reads as a collision. Also: LadderWinShare guarded against the conditional (fractal) geometry path, which fills n from m_fracLegCount while leaving idxList empty - a latent out-of-bounds on a currently-dead path. New log lines: mean label lifespan and effective n on the label-cache line, the required-vs-available horizon per rung, and the grid pair the reconciliation actually measured (its tolerance now scales with the grid skew instead of a flat 5pp). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 01:12:05 -04:00
//--- EFFECTIVE sample, not the raw fire count - the vote's outcomes are overlapping triple-barrier
//--- labels exactly as the member gate's are. See EffectiveSampleSize(); the two gates have to apply
//--- the identical correction or the ensemble becomes the easier one to clear.
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
double precSE = (fired > 0 && chanceP > 0.0 && chanceP < 1.0)
fix(labels): overlapping-label sample correction + horizon cap on the scale ladder Three defects, all surfaced by the 2026-08-17 SP500 H4 run that shipped stop 4.86 / target 9.71 (width 14.57*ATR, horizon 384). 1. EVERY STANDARD ERROR ASSUMED INDEPENDENT SAMPLES. Triple-barrier labels started one per bar overlap by the label's lifespan, so n calls are worth ~n/L independent observations (Lopez de Prado, AFML ch. 4 - sample uniqueness). All three sqrt(p(1-p)/n) sites divided by the RAW count. The tell: the operating point's null-of-the-maximum gate is family-wise and should fire on ~5% of eras under the null. Measured fire rates - PAI 47/73 (64%), ConvLSTM 9/24, LSTM 8/21 (38%), CONV 4/62 (6%). CONV, the only model whose margin distribution admits few bins, sat on the null; the rest cleared a bar that was too low by ~sqrt(L). PAI's deployed threshold consequently alternated between the ENDS of its own range era to era (0.10 -> 0.88 -> 0.86 -> 0.66; coverage 16% <-> 73%). TripleBarrierLabel now records when each label became KNOWABLE - the first winning touch, or both stops, or the timeout - and the prebuild accumulates the mean. EffectiveSampleSize() feeds the operating point, the member deploy gate and the ensemble vote gate. Conservative by construction (n/L is an upper bound on the damage); gates get harder, never easier. 2. THE SCALE LADDER RAN AWAY, again. Horizon scales as swingMedian*sl*tp, and since 4d8cb08 reachability is measured OVER that horizon - so a wider rung buys itself the time that makes it look reachable. Same target -> horizon -> reach -> target loop the excursion window is kept short to avoid; fixing the window confusion reopened it through the other door. It walked 128 -> 256 -> 384 bars and stopped at q90, the widest rung there is, with every rung reading 39-48% against a 20% floor. A floor nothing fails selects nothing. Rungs whose required horizon exceeds BARRIER_HORIZON_MAX are now rejected - the same rule ReportGeometryExpectancyScan already applied. It was printing the shipped pair as CLAMPED and disqualified ('h384!') two lines under the deriver that chose it: two subsystems, one geometry, opposite verdicts. 3. THE RUNG SNAP DESTROYED THE RATIO IT WAS COMPARING. Both legs snapped independently to the coarse first-passage grid, re-rating each candidate: q90 4.86/9.71 -> 5.00/10.00 (2.00), q85 4.07/8.14 -> 5.00/10.00 (IDENTICAL measurement), q75 3.07/6.13 -> 4.00/6.50 (1.63 - a nearer target). So the ladder compared win shares taken at ratios from 1.63 to 2.17 and read the differences as scale. It is why the reach column came out non-monotone in width (q75 48.5% above q90 42.9%). The stop now snaps to its nearest rung in log space and the target follows the ratio off it; the pair actually measured is returned and logged, so a collision reads as a collision. Also: LadderWinShare guarded against the conditional (fractal) geometry path, which fills n from m_fracLegCount while leaving idxList empty - a latent out-of-bounds on a currently-dead path. New log lines: mean label lifespan and effective n on the label-cache line, the required-vs-available horizon per rung, and the grid pair the reconciliation actually measured (its tolerance now scales with the grid skew instead of a flat 5pp). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 01:12:05 -04:00
? 100.0 * MathSqrt(chanceP * (1.0 - chanceP)
/ EffectiveSampleSize((double)fired)) : 0.0;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
double edgeFloorPct = chancePct + EDGE_MIN_SIGMAS * precSE;
//--- Anti-degenerate pair, same intent as the member gate's coverage floor + bothSidesLive: a vote
//--- that fires on almost nothing, or only ever one way, is not a tradeable ensemble however high
//--- its win rate reads. (One-sidedness here is the vote's, not a class-recall measure - a vote
//--- that never goes short IS the always-long model the chance reference already prices in.)
feat(consistency): the five review flaws fixed - training wears the live constraints, the gate wears the policy 1. Labels and the exit simulator go through the broker's stop-distance check: risk/reward widen to SYMBOL_TRADE_STOPS_LEVEL exactly as TCAdjustStops does at order time - the M5/tight-ATR case where live trades ran wider geometry than training measured. Current stops level stands in for history (like the spread); measured quantity, so it does not key the fingerprint. 2. The Intelligent drift verdict moved into RefreshDriftVerdict(), which RESCANS the label cache and now runs at every era end beside RankTiersFromOos - era-cadence instead of waiting for rare full rebuilds. Prints only on change. 3. Session filter is any-broker: sessions defined on their financial centres' civil clocks (London 08-16 Europe/London, NY 08-17 America/New_York, Tokyo 09-18 Asia/Tokyo), converted to UTC by each centre's own computed DST rule (EU last-Sun-Mar/Oct, US 2nd-Sun-Mar/1st-Sun-Nov), then to broker time by the MEASURED server-vs-GMT offset (half-hour brokers included). Windows may wrap midnight in broker time - the interval test handles it. Replaces the EET-hardcoded anchors, which were correct on exactly one broker and got Tokyo wrong by an hour each European summer. 4. The current-session-table-for-history caveat resolved by analysis: the bars bound the error - a too-late assumed close meets no bars (zero error), a too-early one truncates conservatively (<=1h, never optimistic, cannot manufacture edge). Documented at the site. 5. The ensemble deploy gate mirrors the direction policy: blocked-side fires are not fired bars (certified == traded), the zero-skill reference uses only ACHIEVABLE baselines (always-short is not a strategy a long-only book can run), and one-sidedness BY POLICY is not degeneracy - the two-sided requirement applies only when both sides are allowed. Sell predictions keep their other jobs (exit triggers, consensus dilution) untouched. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 12:14:01 -04:00
//--- One-sidedness BY POLICY is not degeneracy: under a one-sided direction policy the vote
//--- CANNOT fire two-sided, and demanding it would refuse deployment forever. The
//--- anti-collapse job survives where it applies - both sides allowed keeps the original
//--- requirement; a one-sided policy asks only that the allowed side actually fires.
bool bothAllowed = (WarriorDirectionAllows(true) && WarriorDirectionAllows(false));
bool twoSided = bothAllowed ? (firedLong > 0 && firedShort > 0) : (fired > 0);
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
bool tradeableOK = measurable && votePrecPct >= 0.0 && twoSided &&
coveragePct >= minCoverPct && votePrecPct > edgeFloorPct;
double coverCredit = 1.0;
if(minCoverPct > 0.0 && coveragePct >= 0.0)
coverCredit = MathMin(1.0, coveragePct / minCoverPct);
double score = (votePrecPct >= 0.0) ? votePrecPct * coverCredit : 0.0;
//--- N for the family-wise correction: every era that COULD have won, mirroring the member gate's
//--- exclusion of eras with nothing to trade.
bool degenerate = (fired <= 0);
if(measurable && !degenerate)
g_ensCandidateEras++;
//--- Same lexicographic ordering as isBetterEra: deployable outranks two-sided outranks score.
bool isBetter = (tradeableOK && !g_ensBestTradeable) ||
(tradeableOK == g_ensBestTradeable && twoSided && !g_ensBestTwoSided) ||
(tradeableOK == g_ensBestTradeable && twoSided == g_ensBestTwoSided &&
!degenerate && score > g_ensBestScore);
if(isBetter)
{
g_ensBestScore = score;
g_ensBestTradeable = tradeableOK;
g_ensBestTwoSided = twoSided;
g_ensBestPrecPct = votePrecPct;
g_ensBestChancePct = chancePct;
g_ensBestCalls = fired;
g_ensBestEra = votedEra;
g_ensErasSinceBest = 0;
g_ensPlateauStage = 0;
EnsembleCommitJointCheckpoint(votedEra);
}
else
g_ensErasSinceBest++;
//--- SHARED PLATEAU LADDER. One counter, one stage, applied to every member at the same era, so
//--- the four nets escalate and finish together instead of drifting into different stages of
//--- different searches. Mechanics per member are unchanged (boosted warm restart + optimizer
//--- reset); only the trigger is collective.
fix(plateau): the IS-error early stop was inert for every ensemble member 15 hours of training, and the stop that exists to END a run announced itself 1,299 consecutive times without ending anything: SP500 ConvLSTM IN-SAMPLE ERROR PLATEAU - not improved in 1297 / 1298 / 1299 eras (best 0.2689, now 0.3269) ... era 1396, 1397, 1398 SP500 LSTM 536 eras SP500 CONV 442 eras SP500 PAI 150 eras XAUUSD HYB 478 eras XAUUSD LSTM 296 eras XAUUSD CONV 366 eras CAUSE: it wrote its decision into m_plateauStage, and EnsembleEraVerdict mirrors the shared ladder onto every member - `mm.m_plateauStage = g_ensPlateauStage` - on EVERY era, purely so each member's status line shows the collective stage. A display mirror was silently overwriting a decision, so the stop re-armed and re-fired the next era, forever. This is the worst possible direction for this particular bug. Every one of those 1,299 eras was scored out of sample and joined the family the deploy gate corrects over (Sidak, g_ensCandidateEras). The stop's entire purpose is to make that family SMALLER; instead the run spent fifteen hours raising its own bar. - m_isErrorPlateaued: a one-way per-member latch, cleared only by a fresh run. Nothing in the ladder may reset it. The stop condition and the two solo deploy conditions read the latch, not the mirrored stage. - The orchestrator combines: EnsembleEraVerdict requires UNANIMITY across participating members (same participation test the era barrier uses, so an excluded or finished member cannot veto). One member still learning can still move the combined vote, and the vote is what the gate certifies. - Fed in as `dueStage = PLATEAU_STAGE_DEPLOY`, NOT written to g_ensPlateauStage. The block that actually ends the run sits under `dueStage > g_ensPlateauStage`, so assigning the stage directly makes that test false and the deploy never happens - the same inert-write shape as the bug being fixed. Caught before committing; raising dueStage carries it through the ladder's own path (warm restarts skipped, family-wise vote test, measurement screen, joint checkpoint) unchanged. - g_ensIsPlateauAnnounced: announce once per run, not once per era. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 13:09:26 -04:00
//--- MEMBERS PUBLISH, THE ORCHESTRATOR COMBINES - the same rule the vote and the live confidence
//--- follow. Each member latches m_isErrorPlateaued when its OWN training error stops improving; this
//--- is the only place allowed to turn that into a collective decision, and it needs UNANIMITY: one
//--- member still learning can still move the combined vote, and the vote is what the gate certifies.
//--- Without this the IS stop was inert for every ensemble member. It wrote m_plateauStage, which the
//--- mirror at the end of this function overwrites every era - so on 2026-08-18 SP500 ConvLSTM
//--- announced the plateau 1,299 times and trained to era 1,398 anyway, adding every one of those eras
//--- to the family the deploy gate must correct over. The stop exists to SHRINK that family.
int plateauedMembers = 0, learningMembers = 0;
for(int pi = 0; pi < ArraySize(g_warriorEnsemble); pi++)
{
CExpertSignalAIBase *pm = g_warriorEnsemble[pi];
if(CheckPointer(pm) == POINTER_INVALID || pm.m_ensembleIndex < 0)
continue;
//--- Same participation test the barrier uses: a member that has finished or been stopped is not
//--- something the rest should wait on, and must not veto the collective stop either.
if(pm.m_trainingComplete || pm.m_trainingStopRequested || !pm.m_isInitialized || pm.m_barrierExcluded)
continue;
if(pm.m_isErrorPlateaued)
plateauedMembers++;
else
learningMembers++;
}
bool allIsPlateaued = (plateauedMembers > 0 && learningMembers == 0);
if(allIsPlateaued && !g_ensIsPlateauAnnounced)
{
g_ensIsPlateauAnnounced = true;
PrintFormat("AI ensemble: EVERY member's IN-SAMPLE error has plateaued (%d participating members)."
" No member is still learning from the data it can see, so more eras cannot find a"
" better vote - they would only enlarge the family the deploy gate corrects over."
" Ending the search on the joint checkpoint at the next era that does not improve it."
" This stop never read an out-of-sample number, which is what makes the smaller family"
" legitimate rather than a peek.", plateauedMembers);
}
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
string ladderNote = "";
if(!isBetter)
{
int dueStage = g_ensErasSinceBest / TrainPlateauPatienceEras();
fix(plateau): the IS-error early stop was inert for every ensemble member 15 hours of training, and the stop that exists to END a run announced itself 1,299 consecutive times without ending anything: SP500 ConvLSTM IN-SAMPLE ERROR PLATEAU - not improved in 1297 / 1298 / 1299 eras (best 0.2689, now 0.3269) ... era 1396, 1397, 1398 SP500 LSTM 536 eras SP500 CONV 442 eras SP500 PAI 150 eras XAUUSD HYB 478 eras XAUUSD LSTM 296 eras XAUUSD CONV 366 eras CAUSE: it wrote its decision into m_plateauStage, and EnsembleEraVerdict mirrors the shared ladder onto every member - `mm.m_plateauStage = g_ensPlateauStage` - on EVERY era, purely so each member's status line shows the collective stage. A display mirror was silently overwriting a decision, so the stop re-armed and re-fired the next era, forever. This is the worst possible direction for this particular bug. Every one of those 1,299 eras was scored out of sample and joined the family the deploy gate corrects over (Sidak, g_ensCandidateEras). The stop's entire purpose is to make that family SMALLER; instead the run spent fifteen hours raising its own bar. - m_isErrorPlateaued: a one-way per-member latch, cleared only by a fresh run. Nothing in the ladder may reset it. The stop condition and the two solo deploy conditions read the latch, not the mirrored stage. - The orchestrator combines: EnsembleEraVerdict requires UNANIMITY across participating members (same participation test the era barrier uses, so an excluded or finished member cannot veto). One member still learning can still move the combined vote, and the vote is what the gate certifies. - Fed in as `dueStage = PLATEAU_STAGE_DEPLOY`, NOT written to g_ensPlateauStage. The block that actually ends the run sits under `dueStage > g_ensPlateauStage`, so assigning the stage directly makes that test false and the deploy never happens - the same inert-write shape as the bug being fixed. Caught before committing; raising dueStage carries it through the ladder's own path (warm restarts skipped, family-wise vote test, measurement screen, joint checkpoint) unchanged. - g_ensIsPlateauAnnounced: announce once per run, not once per era. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 13:09:26 -04:00
//--- FED IN AS A DUE STAGE rather than written straight to g_ensPlateauStage, and the difference is
//--- the whole fix: the block that actually ends the run sits under `dueStage > g_ensPlateauStage`,
//--- so assigning the stage directly makes that test FALSE and the deploy never happens. Raising
//--- dueStage lets the existing ladder carry it through its own path - warm restarts skipped, the
//--- family-wise vote test, the measurement screen, the joint checkpoint - unchanged.
//--- Left inside `if(!isBetter)` deliberately: an era that just produced a better vote produced a
//--- better checkpoint, and ending on the next non-improving era costs one era and keeps it.
if(allIsPlateaued)
dueStage = PLATEAU_STAGE_DEPLOY;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
if(dueStage > g_ensPlateauStage)
{
g_ensPlateauStage = dueStage;
if(g_ensPlateauStage == PLATEAU_STAGE_RESTART || g_ensPlateauStage == PLATEAU_STAGE_ANNEAL)
{
for(int i = 0; i < ArraySize(g_warriorEnsemble); i++)
{
CExpertSignalAIBase *mm = g_warriorEnsemble[i];
if(CheckPointer(mm) == POINTER_INVALID || mm.m_ensembleIndex < 0)
continue;
if(mm.m_trainingComplete || mm.m_trainingStopRequested || !mm.m_isInitialized)
continue;
mm.m_modelEta = mm.m_etaCeiling * PLATEAU_RESTART_BOOST;
mm.m_restartBoostErasLeft = TrainPlateauPatienceEras();
mm.m_plateauStage = g_ensPlateauStage;
if(CheckPointer(mm.Net) != POINTER_INVALID)
mm.Net.ResetOptimizerState();
//--- THIS member is the one still inside Train(), holding eta in a local that would
//--- overwrite m_modelEta on the way out - so its restart has to reach the local too.
if(mm == GetPointer(this))
etaLocal = mm.m_modelEta;
}
ladderNote = StringFormat(" | PLATEAU stage %d: %d eras with no better vote - boosted warm"
" restart on all %d models (learning rate x%.1f, optimizer momentum"
" reset). The joint checkpoint is safe.",
g_ensPlateauStage, g_ensErasSinceBest, members, PLATEAU_RESTART_BOOST);
}
else
if(g_ensPlateauStage >= PLATEAU_STAGE_DEPLOY)
{
//--- EXHAUSTED. Both escapes tried, nothing better found: this is the best vote this
//--- ensemble reaches. Now the gate that matters - has the best-of-N vote survived
//--- having been chosen? (See DEPLOY_FAMILY_WISE_ALPHA: a per-era floor alone opens on
//--- noise with probability 1-(1-a)^N, which is near-certain by era 100.)
double zBest = 0.0, pFam = 1.0;
int nTried = 0;
bool survives = EnsembleSurvivesSelection(zBest, pFam, nTried);
bool haveJoint = true;
for(int i = 0; i < ArraySize(g_warriorEnsemble); i++)
{
CExpertSignalAIBase *mm = g_warriorEnsemble[i];
if(CheckPointer(mm) == POINTER_INVALID || mm.m_ensembleIndex < 0)
continue;
if(mm.m_trainingComplete || mm.m_trainingStopRequested || mm.m_trainingPaused || !mm.m_isInitialized)
continue;
//--- Era-stamped, not just present: a snapshot from an EARLIER era would make the
//--- deployed quartet one that was never measured together (see m_checkpointEra).
if(!mm.m_haveOosCheckpoint || mm.m_checkpointEra != g_ensBestEra)
haveJoint = false;
}
string testNote = StringFormat(" best-of-%d test on the VOTE: edge %.1fpp (%.1f%% vs chance"
" %.1f%%) on %d fired bars = %.2f sigma, family-wise p=%.4f"
" (need <=%.2f)",
nTried, g_ensBestPrecPct - g_ensBestChancePct, g_ensBestPrecPct,
g_ensBestChancePct, g_ensBestCalls, zBest, pFam,
DEPLOY_FAMILY_WISE_ALPHA);
fix(consistency): one geometry authority, one exit authority, and the MI screen finally gets a veto Consistency pass before a fresh deployment. Three places where two systems were choosing the same thing and one of them silently lost. 1. THE GEOMETRY SCAN'S DECISION WAS INERT - measured, not suspected. USDJPY, 2026-08-17: 14:24:12.844 adopting barrier geometry 2:8 ... Relabelling and training on it. 14:24:12.979 triple-barrier labels - stop 1.61*ATR, target 3.21*ATR ... this is what trains It adopted 2:8 and trained on 1.61:3.21. ReportBarrierGeometryScan wrote only m_sl_mode/m_tp_mode, and BarrierMultiples ranks the DERIVED pair ABOVE those ints - so on any model carrying a derived pair (every model with a .cfg, including a fresh one whose weights are gone but whose sidecar survived) the adoption changed nothing. Worse, had it changed something it would have been undone immediately: the adoption sets m_labelCachePrebuilt = false, and that prebuild re-runs DeriveBarrierGeometry at era 0, which overwrites m_derivedSl/TpMult from the excursion quantiles. ONE AUTHORITY: the derived pair, because it is what the labels read, what the deploy gate certifies, what g_Derived*AtrMult places on the live order, and what the .cfg pins across restarts. The scan now writes THAT (floored by MIN_SL_ATR_MULTIPLIER, the same floor DeriveBarrierGeometry applies so the live stop can never be wider than the labelled one), republishes to the bridge immediately rather than at the next era end, and forces the sidecar to be rewritten. m_geometryAdopted latches it so the derive pass the adoption itself triggers cannot overwrite it. The scan outranks the derive for an evidential reason, not an architectural one: its winner cleared a permutation test against the null of the MAXIMUM over every eligible pairing, and it scores the incumbent derived pair as a peer in that same field. The derive is a descriptive quantile read with no significance test attached. BEHAVIOURAL CHANGE, and the reason to flag it before a fresh test: barrier geometry will now actually move when the scan says so. Until today it never did. 2. ONE EXIT AUTHORITY, tied to whose certificate the trade was placed under. CheckClosePosition had two routes. The AI early-exit reads the AI vote undiluted and is exactly what the new exit replay reproduces. The blended route thresholds m_direction, the average over EVERY filter including classic ones whose live votes pass 3 never computes - so it can close a position the certificate never modelled, and no replay can ever check it. When g_DerivedSlAtrMult > 0 the AI's measured geometry is on the order, which means the deploy gate's certificate is the reason the trade exists. In that state the AI now governs the exit and the blended route is suppressed. Classic-only configurations are untouched: there the blended route is the only exit opinion and stays exactly as it was. Nothing moves at the shipped defaults either way (Min_Vote_Close = Disabled). 3. EDGEFINDER, SECOND HALF: THE MEASUREMENT NOW STEERS. The MI suite has always printed its verdicts and then trained the direction target regardless of what they said. That gap IS the difference between this and the EdgeFinder discipline: measure what the market offers, THEN aim. m_dirEvidence is set when EITHER the feature/label mutual information OR the normalised excursion asymmetry clears its block-permuted null - an OR, because the two look for the same thing by different routes and requiring both would reject on the weaker of two independent measurements. Normalised asymmetry specifically, never the raw one, which is the volatility confound. Deploy - solo AND ensemble - now requires it. A run without it still trains, and keeps its checkpoint: the research value is real and the measurement can be wrong. It simply may not go live. Reported separately from the statistical gate because the remedy is different: a failed selection test says train differently, this says look somewhere else. Excursion SIZE keeps clearing where direction does not, and that is a risk-control head rather than an entry signal. For the ensemble the check is per-chart by construction - the MI suite runs once and shares its outcome across members - which is the honest treatment: four models finding nothing between them is not four chances at an edge, it is four fits to the same absent information. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:43:20 -04:00
//--- THE MEASUREMENT SCREEN, applied to the ensemble exactly as to a solo model. The MI
//--- suite runs ONCE per chart and its outcome is shared (see the tune/MI sharing), so
//--- every member on this chart carries the same verdict - checking this member's flag is
//--- checking the chart's. Four models finding nothing between them is not four chances at
//--- an edge; it is four fits to the same absent information.
if(g_ensBestTradeable && haveJoint && survives && !m_dirEvidence)
Print("AI ensemble: DEPLOY REFUSED BY THE MEASUREMENT SCREEN - the combined vote cleared"
" its statistical gate, but neither the feature/label mutual information nor the"
" normalised excursion asymmetry cleared a permutation null on this chart's"
" feature set. The vote is a best-of-N maximum over a search that had no measured"
" directional information to find; clearing the gate on top of that is the"
" family-wise trap this project has hit four times. Checkpoints kept, training"
" untouched - this refuses to go LIVE, nothing else.");
if(g_ensBestTradeable && haveJoint && survives && m_dirEvidence)
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
{
g_ensDeployApproved = true;
Print("AI ensemble: PLATEAU stage " + IntegerToString(PLATEAU_STAGE_DEPLOY) +
" - no better vote for " + IntegerToString(g_ensErasSinceBest) + " eras across " +
IntegerToString(PLATEAU_STAGE_DEPLOY - 1) + " warm restarts." + testNote +
" - CLEARS. Deploying the JOINT checkpoint from era " +
IntegerToString((int)g_ensBestEra) + ": every model reverts to the weights it held"
" at the era whose combined vote scored best, so the ensemble that trades is"
" exactly the one that was measured.");
ladderNote = " | ENSEMBLE DEPLOY APPROVED";
}
else
{
//--- Restart the ladder and keep training, exactly as the solo gate does on a
//--- failed selection test. The era cap stays the backstop.
feat(logs): throttle the settled per-era diagnostics - measured 22MB/9.5h of confirmed-working systems Measured from the journal (2026-08-19): the era deep-dive line (~2KB) plus the excursion verdict, tier re-rank, calibration move, barrier hold and selection-regressed note each printed EVERY era for EVERY member - ~940 eras/member/day - long after the systems they watch were confirmed working. Yesterday's file was 1.3GB (70% of it the news-filter calendar spam the sweep fix already removed). VerboseMode returns as an INPUT (demoted 2026-08-01 for the marketplace; that track is dead since the 2026-08-16 pivot) and gains a second job: false throttles each settled per-era print to eras 0-3 plus every TRAIN_LOG_EVERY_ERAS-th (25 ~= one deep-dive per ~15min per member); true restores the per-era firehose, flippable live. Never throttled: anything that marks a CHANGE - new bests, restores + eta decays, plateau stage transitions, deploy approvals, warnings, errors, the label-cache/adoption one-shots, and the combined-vote gate line (the active system's primary telemetry, still every era). Semantic fixes over blanket gating: - barrier hold now ARMS silently and prints only when the hold outlasts the 2-min report interval - a brief hold every era is the design, the long hold is the watchdog case the line exists for; - the ensemble deploy REFUSAL prints immediately when its reason changes (that is a finding), on cadence when unchanged; - the filtered-view census prints when its RESULT moves (drawn count, or strongest vote by >=2pp) and at least every 10th sweep. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 09:36:15 -04:00
//--- THROTTLED (2026-08-19): the refusal repeated ~450x/day with an unchanged
//--- reason. A CHANGED reason prints immediately - that is a finding; the same
//--- reason keeps the cadence. The APPROVED branch above always prints.
int refusalKey = (!g_ensBestTradeable ? 1 : (!haveJoint ? 2 : 3));
if(refusalKey != m_lastEnsRefusalKey || TrainLogDue())
Print("AI ensemble: PLATEAU stage " + IntegerToString(PLATEAU_STAGE_DEPLOY) + " - " +
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
(!g_ensBestTradeable
? "no era's combined vote ever cleared the deployability floor (fires on at least a"
" quarter as many bars as actually swing, both directions alive, at a win rate"
" above the always-one-way reference by 2 sigma), so there is nothing safe to deploy."
: (!haveJoint
? "the joint checkpoint is incomplete - at least one model has no snapshot of the"
" winning era, so the measured ensemble cannot be reproduced."
: "the best combined vote clears the per-era floor but DOES NOT clear the null of"
" the MAXIMUM over the eras it was chosen from." + testNote +
" A best-of-N this large happens routinely when every era is a noise draw.")) +
" Restarting the ladder and continuing to train; the era cap remains the backstop.");
feat(logs): throttle the settled per-era diagnostics - measured 22MB/9.5h of confirmed-working systems Measured from the journal (2026-08-19): the era deep-dive line (~2KB) plus the excursion verdict, tier re-rank, calibration move, barrier hold and selection-regressed note each printed EVERY era for EVERY member - ~940 eras/member/day - long after the systems they watch were confirmed working. Yesterday's file was 1.3GB (70% of it the news-filter calendar spam the sweep fix already removed). VerboseMode returns as an INPUT (demoted 2026-08-01 for the marketplace; that track is dead since the 2026-08-16 pivot) and gains a second job: false throttles each settled per-era print to eras 0-3 plus every TRAIN_LOG_EVERY_ERAS-th (25 ~= one deep-dive per ~15min per member); true restores the per-era firehose, flippable live. Never throttled: anything that marks a CHANGE - new bests, restores + eta decays, plateau stage transitions, deploy approvals, warnings, errors, the label-cache/adoption one-shots, and the combined-vote gate line (the active system's primary telemetry, still every era). Semantic fixes over blanket gating: - barrier hold now ARMS silently and prints only when the hold outlasts the 2-min report interval - a brief hold every era is the design, the long hold is the watchdog case the line exists for; - the ensemble deploy REFUSAL prints immediately when its reason changes (that is a finding), on cadence when unchanged; - the filtered-view census prints when its RESULT moves (drawn count, or strongest vote by >=2pp) and at least every 10th sweep. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 09:36:15 -04:00
m_lastEnsRefusalKey = refusalKey;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
g_ensErasSinceBest = 0;
g_ensPlateauStage = 0;
for(int i = 0; i < ArraySize(g_warriorEnsemble); i++)
{
CExpertSignalAIBase *mm = g_warriorEnsemble[i];
if(CheckPointer(mm) != POINTER_INVALID && mm.m_ensembleIndex >= 0)
mm.m_plateauStage = 0;
}
ladderNote = " | ladder restarted (gate not cleared)";
}
}
}
}
//--- Mirror the shared ladder onto every member. Their private counters no longer advance (the
//--- ensemble's do), so without this each model's own era line would report "0 eras since best,
//--- stage 0" forever while the ensemble was three stages into its search - a status display that
//--- contradicts the mechanism actually running.
for(int i = 0; i < ArraySize(g_warriorEnsemble); i++)
{
CExpertSignalAIBase *mm = g_warriorEnsemble[i];
if(CheckPointer(mm) == POINTER_INVALID || mm.m_ensembleIndex < 0)
continue;
mm.m_erasSinceBestBalanced = g_ensErasSinceBest;
mm.m_plateauStage = g_ensPlateauStage;
}
feat: implement one-shot pattern-database backfill and enhance accuracy tracking for ensemble models
2026-08-16 21:08:41 -04:00
//--- LIFETIME ACCUMULATION - same cadence as a solo model's m_cumOosTotal (see its increment sites):
//--- every scored era adds the bars the vote fired on and how many paid, monotonically, never reset
//--- per era. `wins`/`fired` above are this era's OOS rows only; the panel reads the running total.
g_ensCumOosTotal += fired;
g_ensCumOosCorrect += wins;
//--- PANEL + JOURNAL. The panel line uses the SAME label and the SAME measurement as a solo model's
//--- ComputeCompoundedAccuracyLine (Expert\AIBase\ChartUI.mqh) - a persisted win-rate over every
//--- called bar, not this era's fired-bar percentage alone - so the two panels read consistently
//--- (user request 2026-08-16: same label, same measurement for both).
string ensAccLine;
if(g_ensCumOosTotal > 0)
{
int winPctLifetime = (int)MathRound(g_ensCumOosCorrect * 100.0 / g_ensCumOosTotal);
//--- THIS ERA alongside the lifetime figure - same reason and same fix as the solo panel's
//--- ComputeCompoundedAccuracyLine (see its "THIS ERA" comment): the lifetime average is diluted
//--- by every fired bar from every prior era, so a real swing this era barely moves it. `wins`/
//--- `fired` above are this era's combined-vote rows only.
string thisEra = (fired > 0) ? StringFormat(", this era %d%%", (int)MathRound(votePrecPct)) : "";
ensAccLine = StringFormat("Buy/Sell calls correct: %d%% (unseen data%s%s)", winPctLifetime,
(bePct >= 0 ? StringFormat(", need %d%%", bePct) : ""), thisEra);
}
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
else
feat: implement one-shot pattern-database backfill and enhance accuracy tracking for ensemble models
2026-08-16 21:08:41 -04:00
ensAccLine = (g_ensCandidateEras > 0) ? "Buy/Sell calls correct: no directional calls yet"
: "Buy/Sell calls correct: measuring...";
g_ensembleVoteLine = StringFormat("%s (era %d, %d models%s)", ensAccLine, (int)votedEra, members,
(g_ensDeployApproved ? ", DEPLOYING" : (tradeableOK ? ", deployable" : "")));
feat(ensemble): per-NN inputs replace the preset selector - the meta head becomes the vote's gate User design (2026-08-19): 'remove the enum menu that selects neural networks... individual inputs for every NN just like classic signals... the META NN should be integrated into the voting decision pipeline when enabled... as a bonus meta labelling is applied to enabled NNs.' - AI_CHOICE is GONE (tombstoned per the stale-.set doctrine). Use_MLP/Use_CONV/Use_LSTM/ Use_CONVLSTM are ordinary bools like the classic votes; the ensemble arithmetic adapts to any subset because the consensus divisor is the enabled capable weight. Two or more enabled = ensemble (|ENS1 token + joint gate, exactly the old AI_HYBRID fingerprints, so existing weight files keep loading); one = the old solo preset; none = classic-only. - Use_MetaLabeling un-couples META from the direction NNs (the old selector made them mutually exclusive). S3 ships: CSignalMETA::LiveMetaGate scores each vote-cleared entry (shared window at bar 1 + proposal descriptor: side, net vote, live geometry, spread/ATR; pattern one-hot ZEROED - ranking, not calibrated probability, documented in the body) and vetoes below the cost-adjusted break-even. Entries only; fail-open everywhere, loudly. - COEXISTENCE HAZARDS closed: VoteCapableWeight()=0 and ProspectiveVote()=false for the meta target - solo-only until today, a trained META would otherwise sit in the consensus divisor as a permanent abstainer and shrink every vote by its module weight. - CERTIFIED == TRADED: the ensemble era verdict replays the identical veto through the same g_warriorMetaGate pointer over its OOS fired bars (bar re-resolved from the row's own time; fail-open counted as fires and reported: 'metaGate: N approved, M vetoed, K unscored'). The overlay deliberately does NOT replay it (veto-filter-in-replay class, calendar-cliff precedent) - documented at the sweep site. Solo charts' own gate does not model the veto - the standing solo-gate caveat, documented at the input. - DB continuity: the pattern/journal DB fingerprint's first slot was (int)AIType; DbLegacyAiSlot() maps every legacy-expressible config to its OLD value (new 2-3 member subsets get 100+bitmask, outside the legacy range) so no existing database re-keys. filterID becomes the enabled roster via one EnabledNNSummary(). - HUD: the meta line shows the gate (armed/(trn), last P vs BE, ok/veto tally); the armed/disarmed announcement fires on state change via one latch (MetaGateArmedNow), not only when an entry happens to be proposed. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 13:01:02 -04:00
//--- Present only when a meta head is attached and at least one fired bar reached it - the
//--- unscored count is the honesty term (bars the gate could not score are certified as fires
//--- because live they would trade ungated).
string metaNote = (g_warriorMetaGate != NULL && (metaOk + metaVetoed + metaOpen) > 0)
? StringFormat(" | metaGate: %d approved, %d vetoed, %d unscored(open)",
metaOk, metaVetoed, metaOpen)
: "";
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
Print(StringFormat("AI ensemble: combined-vote era %d - %d models, %d shared OOS bars, %d fired at"
" vote>=%.0f%% (%.1f%% coverage, floor %.1f%%), win %s vs chance %.1f%% (needs"
fix(gate): the ensemble gate certified a vote the EA never casts g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on the same criterion the live trade does". It did not. Two independent mismatches, both silent: CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100 number straight off the softmax head. Live contributes m_weight x the tier's pattern weight, and BOTH of those are rewritten from the signal DB by UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share an axis and nothing relates them, so the same bar was one number to the gate and a different one to the order path. Same shape as the 2026-08-09 geometry incident: certified on one game, paid on another. DENOMINATOR. The gate divided by the member count, so an abstaining member pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero contribution in BOTH the sum and the count (`if(direction == 0) continue;` before `number++`) - live is a mean over VOTERS. The gate was therefore scoring a strictly more agreement-heavy set of bars than the EA trades. The contribution hook's own comment asserted the opposite ("abstentions dilute the average exactly as they do in the live vote"), while the AI_CHOICE enum 20 lines away correctly documented union semantics. LiveVoteContribution() is now the single definition of "what this member votes", called from the gate; the live path reaches the same arithmetic through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually voted, separately from who evaluated the bar, because those are the divisor and the shared-population test respectively. ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar wrapper - the OOS scan holds the scanned bar's decision in a local, and dPrevSignal is a different bar. NOT changed, deliberately: the gate still does not model live NMS declustering, and the per-member solo gate still scores every directional call rather than threshold-clearing ones. Both are selection-metric changes and this codebase has twice been bitten by switching one blind. Also corrects a stale paragraph in m_pattern_0's declaration block quoting 80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has been since the confidence floor and alternation gate were removed; the block carried both tables at once, and the dead one was quoted back as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
" >%.1f%% at %d sigma)%s -> score %.1f%s%s. The vote that actually trades: each"
" member's DB-ranked tier weight x module weight, averaged over the members that"
" VOTED (abstentions excluded, as live), graded on target-before-stop.",
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
(int)votedEra, members, shared, fired, g_ensembleVoteThreshold,
coveragePct, minCoverPct,
(fired > 0 ? StringFormat("%.1f%%", votePrecPct) : "n/a"), chancePct, edgeFloorPct,
(int)EDGE_MIN_SIGMAS, (tradeableOK ? " DEPLOYABLE" : ""), score,
(isBetter ? StringFormat(" <-- NEW BEST, joint checkpoint captured (era %d)", (int)votedEra)
: StringFormat(" (best %.1f at era %d, %d eras ago)", g_ensBestScore,
(int)g_ensBestEra, g_ensErasSinceBest)),
feat(ensemble): per-NN inputs replace the preset selector - the meta head becomes the vote's gate User design (2026-08-19): 'remove the enum menu that selects neural networks... individual inputs for every NN just like classic signals... the META NN should be integrated into the voting decision pipeline when enabled... as a bonus meta labelling is applied to enabled NNs.' - AI_CHOICE is GONE (tombstoned per the stale-.set doctrine). Use_MLP/Use_CONV/Use_LSTM/ Use_CONVLSTM are ordinary bools like the classic votes; the ensemble arithmetic adapts to any subset because the consensus divisor is the enabled capable weight. Two or more enabled = ensemble (|ENS1 token + joint gate, exactly the old AI_HYBRID fingerprints, so existing weight files keep loading); one = the old solo preset; none = classic-only. - Use_MetaLabeling un-couples META from the direction NNs (the old selector made them mutually exclusive). S3 ships: CSignalMETA::LiveMetaGate scores each vote-cleared entry (shared window at bar 1 + proposal descriptor: side, net vote, live geometry, spread/ATR; pattern one-hot ZEROED - ranking, not calibrated probability, documented in the body) and vetoes below the cost-adjusted break-even. Entries only; fail-open everywhere, loudly. - COEXISTENCE HAZARDS closed: VoteCapableWeight()=0 and ProspectiveVote()=false for the meta target - solo-only until today, a trained META would otherwise sit in the consensus divisor as a permanent abstainer and shrink every vote by its module weight. - CERTIFIED == TRADED: the ensemble era verdict replays the identical veto through the same g_warriorMetaGate pointer over its OOS fired bars (bar re-resolved from the row's own time; fail-open counted as fires and reported: 'metaGate: N approved, M vetoed, K unscored'). The overlay deliberately does NOT replay it (veto-filter-in-replay class, calendar-cliff precedent) - documented at the sweep site. Solo charts' own gate does not model the veto - the standing solo-gate caveat, documented at the input. - DB continuity: the pattern/journal DB fingerprint's first slot was (int)AIType; DbLegacyAiSlot() maps every legacy-expressible config to its OLD value (new 2-3 member subsets get 100+bitmask, outside the legacy range) so no existing database re-keys. filterID becomes the enabled roster via one EnabledNNSummary(). - HUD: the meta line shows the gate (armed/(trn), last P vs BE, ok/veto tally); the armed/disarmed announcement fires on state change via one latch (MetaGateArmedNow), not only when an entry happens to be proposed. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 13:01:02 -04:00
ladderNote + metaNote));
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
}
//+------------------------------------------------------------------+
//| Per-member era-end hook: mark this member done for the era and, |
//| when it is the last one, run the verdict above. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::EnsembleOosPassComplete(const long votedEra, double &etaLocal)
{
if(!m_ensembleMember || m_ensembleIndex < 0)
return;
//--- The rows were stamped during pass 3, BEFORE this member incremented its era counter, so the
//--- buffer's era is the era that just finished. A mismatch means this member contributed nothing
//--- to the current buffer (no OOS bars scored this era) - it cannot be counted as having read the
//--- vote, or the verdict would be taken on a subset that silently excludes it.
if(g_ensVoteEra != votedEra)
return;
g_ensVoteDoneMask |= (1 << m_ensembleIndex);
int need = 0;
for(int i = 0; i < ArraySize(g_warriorEnsemble); i++)
{
CExpertSignalAIBase *mm = g_warriorEnsemble[i];
if(CheckPointer(mm) == POINTER_INVALID || mm.m_ensembleIndex < 0)
continue;
if(mm.m_trainingComplete || mm.m_trainingStopRequested || mm.m_trainingPaused || !mm.m_isInitialized)
continue;
need |= (1 << mm.m_ensembleIndex);
}
if(need == 0 || (g_ensVoteDoneMask & need) != need)
return;
//--- Idempotence: one verdict per era, whatever order the members arrive in.
if(g_ensLastVerdictEra == votedEra)
return;
g_ensLastVerdictEra = votedEra;
EnsembleEraVerdict(need, votedEra, etaLocal);
}
//+------------------------------------------------------------------+
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
//| Log the selection-gate verdict for a deploy the gate does NOT |
//| block - the era-cap path and the panel's Deploy button, both of |
//| which are explicit operator decisions and stay that way. The point |
//| is that "I chose to ship this" and "this cleared the bar" should |
//| never be confusable in the log afterwards. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::ReportSelectionGateVerdict(string context)
{
double z = 0.0, pFam = 1.0;
int nTried = 0;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//--- ENSEMBLE: report the gate that actually governs this model. Quoting the member's own
//--- best-of-N here would answer a question nobody asked - the member never deploys alone, and a
//--- member-level "CLEARS" next to a vote that did not is precisely the confusion this function
//--- exists to prevent.
if(m_ensembleMember)
{
bool okEns = EnsembleSurvivesSelection(z, pFam, nTried);
if(g_ensBestCalls <= 0)
{
Print(ID + ": " + context + " - the ENSEMBLE selection gate cannot be evaluated (no era's"
" combined vote has been ranked yet). Treat this ensemble as unvalidated.");
return;
}
Print(ID + ": " + context + " - ENSEMBLE best-of-" + IntegerToString(nTried) + " test on the"
" combined VOTE: edge " + DoubleToString(g_ensBestPrecPct - g_ensBestChancePct, 1) + "pp (" +
DoubleToString(g_ensBestPrecPct, 1) + "% vs chance " + DoubleToString(g_ensBestChancePct, 1) +
"%) on " + IntegerToString(g_ensBestCalls) + " fired bars = " + DoubleToString(z, 2) +
" sigma, family-wise p=" + DoubleToString(pFam, 4) + " (need <=" +
DoubleToString(DEPLOY_FAMILY_WISE_ALPHA, 2) + ") - " +
(okEns ? "CLEARS."
: "DOES NOT CLEAR. A maximum this size arises routinely when every era is a noise"
" draw, so this ensemble is being deployed on operator authority, NOT on measured"
" evidence of an edge."));
return;
}
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
bool ok = BestCheckpointSurvivesSelection(z, pFam, nTried);
if(m_bestDirCalls <= 0)
{
Print(ID + ": " + context + " - selection gate cannot be evaluated (no ranked checkpoint with"
" directional calls). Treat this model as unvalidated.");
return;
}
Print(ID + ": " + context + " - best-of-" + IntegerToString(nTried) + " selection test: edge " +
DoubleToString(m_bestDirPrecPct - m_bestChancePrecPct, 1) + "pp (" +
DoubleToString(m_bestDirPrecPct, 1) + "% vs chance " + DoubleToString(m_bestChancePrecPct, 1) +
"%) on " + IntegerToString(m_bestDirCalls) + " directional calls = " + DoubleToString(z, 2) +
" sigma, family-wise p=" + DoubleToString(pFam, 4) + " (need <=" +
DoubleToString(DEPLOY_FAMILY_WISE_ALPHA, 2) + ") - " +
(ok ? "CLEARS."
: "DOES NOT CLEAR. A maximum this size arises routinely when every era is a noise draw, so"
" this model is being deployed on operator authority, NOT on measured evidence of an edge."));
}
//+------------------------------------------------------------------+
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//| Training and Signal Methods |
//+------------------------------------------------------------------+
fix: a restart no longer loses the measured geometry or the training window Terminal restart, 22:25: all four resumed models sat on empty windows with enum 2:6 barriers. Three interlocking causes, all visible in one log excerpt: 1) THE PRE-SCAN WINDOW WAS SIZED BY THE SAVED WATERMARK. A resumed model's dtStudied sits at its last studied bar, so Bars(dtStudied, now) ~ 0 and the resumed-model MI pre-scan built a zero-bar "complete" label cache - logged as "Buy: 0 | Sell: 0 | Neutral: 0". Train()'s own era start RESETS dtStudied to the training-window rule before computing its window; the pre-scan did not. The rule is now factored into TrainWindowStart() and both use it. The scan also refuses to arm before SERIES_SYNCHRONIZED (it ran in the same second as OnInit), and deployed models keep their watermark - for them it gates inference recency, not a training window. 2) THE HORIZON LATCHED ON AN INDICATOR WARM-UP. ComputeBarrierHorizonBars ran against a ZigZag with 0 calculated legs, fell back, and EnsureBarrierHorizon latched fallback(32) x slMult x tpMult = 384 for the process lifetime. A leg-starved horizon is now PROVISIONAL: re-resolved on the next rebuild, the label cache wiped if it moved (labels from two horizons answer different questions), and the geometry deriver refuses to run from it - a pair derived over a warm-up window would get PINNED. 3) THE DERIVED GEOMETRY WAS NEVER PERSISTED. The .cfg is written at model creation and at weights-reset - both BEFORE era 0 derives - so the measured pair lived only in memory: every restart read back zeros, adopted nothing, fell back to the enum barriers, and the era-0-only gate meant a resumed model could NEVER re-derive. A full day of training on 3.33/1.62 resumed as 2:6. Now: the settled pair is pinned to the .cfg the moment derivation completes (one-shot, atomic write), and the derive gate accepts any model with no pinned pair, not just era 0 - mid-run stability is carried by m_geometryDerived itself, which never allows a second derivation. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 22:40:43 -04:00
//+------------------------------------------------------------------+
//| Where the TRAINING window starts: ALL available history, floored |
//| by MinTrainYear. The StudyPeriods input this replaced could only |
//| ever throw data away: the signal is weak and the directional |
//| classes are rare, so every extra year is more of the minority |
//| class, and the honest generalization read comes from the OOS |
//| holdout rather than from withholding history. MinTrainYear |
//| survives because it answers a different question - excluding a |
//| broker's dubious pre-history - not "how much". |
//| Ordering: SERIES_FIRSTDATE is the floor of what EXISTS, |
//| MinTrainYear the floor of what is TRUSTED; the window starts at |
//| whichever is later. |
//| Shared by Train()'s era start and StartLabelCachePrebuild(), so |
//| the pre-scan and the era loop can never disagree about what "the |
//| window" means - the saved dtStudied watermark is NOT an input |
//| here, which is the point (see the call site in Train()). |
//+------------------------------------------------------------------+
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
//+------------------------------------------------------------------+
fix: prebuild and era sized different windows; diag: Train() names its branch TWO things, one incident. 1) THE BUG I SHIPPED IN 0c85c54. m_tuneStartTrainBar is declared, initialised to 0, and NEVER ASSIGNED - the assignment existed before the God-class split and the split dropped it, leaving a dead member. Harmless while nothing read it; a real defect the moment 0c85c54 made StartLabelCachePrebuild() reset dtStudied from it. Train() then computed the window as max(StartTrainBar, floor) while the prebuild computed max(0, floor), where StartTrainBar is the non-zero datetime OnChartEventHandler passes through from the "New Bar" event. The two therefore disagreed about `bars`, so EnsureBarCachesCapacity() saw a changed size at era start, wiped the caches, and re-armed a full 38k-bar prebuild - instead of training. Restored the assignment so both sides evaluate the identical expression. 2) THE REASON IT TOOK ALL NIGHT TO FIND. Train() is a state machine with six early-return branches above the era loop and every one of them is silent. Four charts burned a core each for 15 minutes with an empty journal: the pass heartbeats (694b756) proved the era loop was never reached, no prebuild completion line appeared either, and nothing external can see inside a single MQL5 thread - per-thread CPU says "busy", file writes say nothing, and the VPS has no debugger. That is an undiagnosable state, and it is the thing to fix, not just the bug of the day. ReportTrainStall() now names the branch Train() is taking whenever no era has completed for 3 minutes, at most once a minute per signal, with the state that decides the branch: run/prebuild/simOos/resume flags, era, dtStudied, and - for the cache-invalidation branch specifically - BOTH bar counts, since two sizings disagreeing is exactly what re-arms the prebuild forever. Silent on a healthy run: an era completing resets the clock. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:32:08 -04:00
//| Names the Train() branch being taken while no era has completed |
//| for a long time. Silent on a healthy run (an era ends, the clock |
//| resets); at most one line per 60s per signal once stalled. |
//| See m_lastEraCompleteTick for the incident that forced this. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::ReportTrainStall(const string branch)
{
const uint STALL_AFTER_MS = 180000; // 3 min: ~2x the slowest healthy era seen on this config
const uint STALL_REPORT_INTERVAL = 60000;
uint nowTick = GetTickCount();
//--- First call ever: adopt now as the baseline rather than reporting instantly against tick 0.
if(m_lastEraCompleteTick == 0)
{
m_lastEraCompleteTick = nowTick;
return;
}
uint since = nowTick - m_lastEraCompleteTick;
if(since < STALL_AFTER_MS)
return;
if(m_lastStallReportTick != 0 && nowTick - m_lastStallReportTick < STALL_REPORT_INTERVAL)
return;
m_lastStallReportTick = nowTick;
PrintFormat("%s: TRAIN STALL - no era has completed for %.0fs and Train() is taking the '%s' branch"
" | era %d | runActive=%s prebuildActive=%s cachePrebuilt=%s simOos=%s eraResume=%s"
" paused=%s stopReq=%s | labelCacheBars=%d anchor=%s dtStudied=%s",
ID, since / 1000.0, branch, (int)m_eraCount,
m_trainRunActive ? "Y" : "N", m_labelPrebuildActive ? "Y" : "N",
m_labelCachePrebuilt ? "Y" : "N", m_simOosRunActive ? "Y" : "N",
m_eraResumePending ? "Y" : "N", m_trainingPaused ? "Y" : "N",
m_trainingStopRequested ? "Y" : "N",
m_labelCacheBars, TimeToString(m_labelCacheAnchorTime), TimeToString(dtStudied));
}
//+------------------------------------------------------------------+
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
//| Speaks ONLY when an era is genuinely slow: nothing for the first |
//| 60 seconds of an era, at most 6 lines after that, one per 4096 |
//| processed items. Reports where the time actually went, split into |
//| the two candidate costs and the remainder, because "the era is |
//| slow" without the split is exactly the undiagnosable state the |
//| 2026-08-10 restart produced (see the member declarations). |
//+------------------------------------------------------------------+
fix: the panel showed "100%" for the whole of pass 1 The simple panel derived its percentage from pass 2's counters: (m_isTrainCursor+1) / max(m_isTrainQueueCount,1). During pass 1 those are 0 and 0, so the expression is (0+1)/max(0,1) = 100%. An era spends its first pass scanning ~38k bars - minutes of work - and the panel reported that phase as finished the entire time. Observed by the user as "started learning at 100% of their era and are stuck there", and it actively misled the diagnosis: the one number on screen said the opposite of what was happening. The UI cannot fix this on its own - it can see pass 2's counters but has no way to know which pass owns them. So each pass now PUBLISHES its own progress and a short phase name through TrainHeartbeat (which every pass already calls per item), and the panel just displays them: "learning (era 45, scan 34%)". Published before the heartbeat's 4096-item journal gate, so the panel updates continuously while the journal stays quiet. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:22:29 -04:00
void CExpertSignalAIBase::TrainHeartbeat(const string tag, int done, int total, const string shortLabel)
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
{
fix: the panel showed "100%" for the whole of pass 1 The simple panel derived its percentage from pass 2's counters: (m_isTrainCursor+1) / max(m_isTrainQueueCount,1). During pass 1 those are 0 and 0, so the expression is (0+1)/max(0,1) = 100%. An era spends its first pass scanning ~38k bars - minutes of work - and the panel reported that phase as finished the entire time. Observed by the user as "started learning at 100% of their era and are stuck there", and it actively misled the diagnosis: the one number on screen said the opposite of what was happening. The UI cannot fix this on its own - it can see pass 2's counters but has no way to know which pass owns them. So each pass now PUBLISHES its own progress and a short phase name through TrainHeartbeat (which every pass already calls per item), and the panel just displays them: "learning (era 45, scan 34%)". Published before the heartbeat's 4096-item journal gate, so the panel updates continuously while the journal stays quiet. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:22:29 -04:00
//--- Panel progress is published on EVERY call, before the 4096-item gate below: the gate exists to
//--- keep the JOURNAL quiet, and applying it to the panel too would leave the display frozen between
//--- boundaries. Two assignments, no formatting - cheap enough for a per-item path.
m_passLabel = shortLabel;
m_passProgressPct = (total > 0) ? (int)MathMin(100.0, 100.0 * done / total) : 0;
diag: the heartbeat could be outrun by the condition it watched for It fired only on 4096-item boundaries once an era had already run 60s. Those boundaries are all crossed in the first few chunks of pass 1, so an era that became slow AFTER them printed nothing at all - which is precisely what happened: 20 minutes, four pegged cores, zero heartbeats. I read that silence as "the era loop is never reached" and went looking for a wedge above it. The silence may simply have meant "past the last boundary". A diagnostic whose trigger can be outrun by the condition it watches for is worse than no diagnostic, because it produces confident wrong conclusions. Now time-gated: checked every 256 items (the mask only keeps GetTickCount off the hot path), prints when the era has run >60s and >30s since the last line, up to 12 per era. Progress/phase for the panel is still published on every call, before any gate. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:44:03 -04:00
//--- TIME-gated, not item-gated. The first version fired only on 4096-item boundaries once the era
//--- had already run 60s - and those boundaries are all crossed in the first few chunks, so a run
//--- that got slow AFTER them printed nothing at all. That is exactly what happened on 2026-08-10:
//--- 20 minutes, four pegged cores, zero heartbeats, and the silence was read as "the era loop is
//--- never reached" when it may simply have been past its last boundary. A diagnostic whose
//--- trigger can be outrun by the condition it watches for is worse than none - it produces
//--- confident wrong conclusions. The 255-item mask only keeps GetTickCount() off the hot path.
if((done & 255) != 0)
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
return;
diag: the heartbeat could be outrun by the condition it watched for It fired only on 4096-item boundaries once an era had already run 60s. Those boundaries are all crossed in the first few chunks of pass 1, so an era that became slow AFTER them printed nothing at all - which is precisely what happened: 20 minutes, four pegged cores, zero heartbeats. I read that silence as "the era loop is never reached" and went looking for a wedge above it. The silence may simply have meant "past the last boundary". A diagnostic whose trigger can be outrun by the condition it watches for is worse than no diagnostic, because it produces confident wrong conclusions. Now time-gated: checked every 256 items (the mask only keeps GetTickCount off the hot path), prints when the era has run >60s and >30s since the last line, up to 12 per era. Progress/phase for the panel is still published on every call, before any gate. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:44:03 -04:00
uint nowTick = GetTickCount();
uint elapsedMs = nowTick - m_eraStartTick;
if(elapsedMs < 60000 || m_passHeartbeatPrints >= 12)
return;
if(m_lastHeartbeatTick != 0 && nowTick - m_lastHeartbeatTick < 30000)
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
return;
diag: the heartbeat could be outrun by the condition it watched for It fired only on 4096-item boundaries once an era had already run 60s. Those boundaries are all crossed in the first few chunks of pass 1, so an era that became slow AFTER them printed nothing at all - which is precisely what happened: 20 minutes, four pegged cores, zero heartbeats. I read that silence as "the era loop is never reached" and went looking for a wedge above it. The silence may simply have meant "past the last boundary". A diagnostic whose trigger can be outrun by the condition it watches for is worse than no diagnostic, because it produces confident wrong conclusions. Now time-gated: checked every 256 items (the mask only keeps GetTickCount off the hot path), prints when the era has run >60s and >30s since the last line, up to 12 per era. Progress/phase for the panel is still published on every call, before any gate. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:44:03 -04:00
m_lastHeartbeatTick = nowTick;
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
m_passHeartbeatPrints++;
double featS = (double)m_passFeatUs / 1000000.0;
double netS = (double)m_passNetUs / 1000000.0;
PrintFormat("%s: SLOW ERA heartbeat - %s %d of %d after %.0fs | feature windows %.1fs | net fwd/back %.1fs | everything else %.1fs",
ID, tag, done, total, elapsedMs / 1000.0, featS, netS,
MathMax(elapsedMs / 1000.0 - featS - netS, 0.0));
}
//+------------------------------------------------------------------+
fix: a restart no longer loses the measured geometry or the training window Terminal restart, 22:25: all four resumed models sat on empty windows with enum 2:6 barriers. Three interlocking causes, all visible in one log excerpt: 1) THE PRE-SCAN WINDOW WAS SIZED BY THE SAVED WATERMARK. A resumed model's dtStudied sits at its last studied bar, so Bars(dtStudied, now) ~ 0 and the resumed-model MI pre-scan built a zero-bar "complete" label cache - logged as "Buy: 0 | Sell: 0 | Neutral: 0". Train()'s own era start RESETS dtStudied to the training-window rule before computing its window; the pre-scan did not. The rule is now factored into TrainWindowStart() and both use it. The scan also refuses to arm before SERIES_SYNCHRONIZED (it ran in the same second as OnInit), and deployed models keep their watermark - for them it gates inference recency, not a training window. 2) THE HORIZON LATCHED ON AN INDICATOR WARM-UP. ComputeBarrierHorizonBars ran against a ZigZag with 0 calculated legs, fell back, and EnsureBarrierHorizon latched fallback(32) x slMult x tpMult = 384 for the process lifetime. A leg-starved horizon is now PROVISIONAL: re-resolved on the next rebuild, the label cache wiped if it moved (labels from two horizons answer different questions), and the geometry deriver refuses to run from it - a pair derived over a warm-up window would get PINNED. 3) THE DERIVED GEOMETRY WAS NEVER PERSISTED. The .cfg is written at model creation and at weights-reset - both BEFORE era 0 derives - so the measured pair lived only in memory: every restart read back zeros, adopted nothing, fell back to the enum barriers, and the era-0-only gate meant a resumed model could NEVER re-derive. A full day of training on 3.33/1.62 resumed as 2:6. Now: the settled pair is pinned to the .cfg the moment derivation completes (one-shot, atomic write), and the derive gate accepts any model with no pinned pair, not just era 0 - mid-run stability is carried by m_geometryDerived itself, which never allows a second derivation. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 22:40:43 -04:00
datetime CExpertSignalAIBase::TrainWindowStart(datetime startTrainBar)
{
datetime firstAvailableBar = (datetime)SeriesInfoInteger(m_symbol.Name(), PERIOD_CURRENT, SERIES_FIRSTDATE);
MqlDateTime floor_time;
TimeCurrent(floor_time);
floor_time.year = m_minTrainYear;
floor_time.mon = 1;
floor_time.day = 1;
floor_time.hour = 0;
floor_time.min = 0;
floor_time.sec = 0;
datetime st_time = StructToTime(floor_time);
if(firstAvailableBar > st_time)
st_time = firstAvailableBar;
return MathMax(startTrainBar, st_time);
}
//+------------------------------------------------------------------+
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
void CExpertSignalAIBase::Train(datetime StartTrainBar = 0)
{
fix: improve error messages and suppress false sharing-violation logs - BufferDouble: replace hardcoded "DirectML/CPU-DLL" with dynamic backend name and add buffer index/element count to all error prints for easier debugging. - NetPersistence: distinguish missing file from transient lock by probing FileIsExist before logging, eliminating false "sharing violation" warnings when no saved model exists on first run.
2026-08-02 01:09:18 -04:00
//--- One-shot latch so a failing forward pass reports itself ONCE per call instead of once per
//--- sample. CNet::feedForward's return value used to be discarded at all three call sites below,
//--- which is how the 2026-08-02 run spent a whole era backpropagating against a batch-norm layer
//--- whose device-side output had frozen: the only trace was 13,776 identical BufferWrite lines
//--- from three frames deeper, and nothing said training was still running on top of them.
bool forwardFailureReported = false;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
const int STABILITY_WINDOW = 3; // consecutive eras the OOS accuracy must hold steady for
const double STABILITY_TOLERANCE = 2.0; // max spread (percentage points) across that window
// Max wall-clock work per call before yielding - see m_trainRunActive's declaration comment for
// why chunking exists at all. TuneIndicatorsAndTrain()/Train() only run ONCE per dispatched
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
// "New Bar" custom chart event (see OnChartEventHandler - this instance's m_studyEventId calls it
// exactly once, then
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// clears bEventStudy so ScheduleTrainingIfNeeded() can arm the next one), so the real throughput
// ceiling in practice is however fast MT5 itself pumps/dispatches that custom event - NOT this
// constant. Raising the OnTimer interval (5s->250ms) had ~zero effect for exactly that reason:
// ticks/chart events were already redispatching far more often than the timer alone would. Since
// per-event dispatch overhead is roughly fixed, doing more compute per event (fewer, larger
// chunks) cuts wall-clock training time roughly in proportion, but MT5 has only this one thread -
// the panel/chart can only respond to input in the gap between chunks, so 500ms made it feel
// unresponsive unless clicks landed in that narrow window. Lowered back to 120ms to keep the UI
// reactive. Raised to 200ms 2026-07-26 (throughput became the bigger complaint, as flagged above) -
// a deliberate middle ground between the reactive-but-slow 120ms and the previously-rejected 500ms,
// not a return to that. Watch panel drag/click feel after this change; back off toward 120ms if it
// regresses, or raise further only in small steps if it doesn't.
fix(ui): unique chart tag, product-grade panel, responsive under load Three separate reports from one deploy. 1. CONV, LSTM and HYBRID all came back tagged [4109]. The weights fingerprint omits the topology type on purpose - the file path already separates it (State\CONV\ vs State\LSTM\ vs State\HYB\) and hashing a value that is constant within a folder buys nothing while re-keying every trained model into a forced retrain. So the files were never at risk, but the tag could not do its one job. Prefixing the short id makes it unique on the display side only; the hex half still greps straight to the .nnw inside the folder the prefix names. 2. The default panel read like a training console. Six lines down to three, each answering a question an owner actually has. The deploy internals (best score, eras-since-best, ladder stage) were developer diagnostics describing a recall floor that no longer decides anything, and were already in the era-end journal line. In-sample accuracy left the panel too: it grades the model on bars it trained on, so it always flatters, and showing it beside the honest number invites reading the wrong one. New compile-time DebuggingMode constant - deliberately not an input - carries the IS/OOS pair and the resolved model path into the journal instead. No extra Inputs row, no extra Market description line, no user-reachable firehose. 3. Panel drag and buttons stuttered under training load, exactly as the 2026-07-26 note raising the chunk budget to 200ms warned they might. Backed off to the documented 120ms - worst-case click latency is that budget - and the derived topology (~292k weights to ~29k) makes the throughput this costs far cheaper than when that note was written. Also halved the panel redraw rate to 2.5 Hz: ChartRedraw repaints the whole chart, so its cost scales with accumulated arrows, and 5 Hz was the larger half of the stutter. Era-end still force-refreshes. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 09:05:58 -04:00
// 2026-07-30: it regressed, exactly as that warning anticipated - the panel drags stickily and
// buttons miss clicks under load, because 200ms is the worst-case latency between a click landing
// and this thread being free to notice it. Backing off to the documented 120ms. The throughput this
// costs is a far smaller sacrifice than it was when the note above was written: the derived topology
// cut the network from ~292k weights to ~29k (see ComputeFirstLayerWidth), so an era is a fraction
// of the work it used to be and the fixed per-dispatch overhead the note worried about is now a
// correspondingly smaller share of it. Responsiveness is worth more than the remainder.
fix(ensemble): shared MI diagnostics + divided chunk budget - warm-up and panel responsiveness Two user-reported ensemble regressions, one cause each: - "getting ready is very long": every member ran the full MI diagnostic suite (headline MI, positive control, alignment, lag profile, geometry scan + winner test - ~200 permuted draws per line) on IDENTICAL features and labels, reporting the same numbers four times. First member runs it, the rest adopt with one log line. Documented caveat: if the geometry scan ever ADOPTS a winner under its gate (it never has), the adoption becomes donor-only and the gate must be revisited. - "panel not responsive": four members chunks queue back-to-back on the one chart thread - 4 x 120ms = 480ms worst-case click latency, the exact regime the 200ms note in Training.mqh already documents as broken. Ensemble members now use a 30ms chunk budget, restoring solo UI latency at slightly higher dispatch overhead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 17:37:10 -04:00
//--- ENSEMBLE: four members share the one chart thread and their chunks queue back-to-back, so the
//--- worst-case latency between a panel click and a free thread is members x budget - 4 x 120ms =
//--- 480ms, which is exactly the "drags stickily, buttons miss clicks" regime the 200ms note above
//--- documents (user-reported on the first ensemble runs, 2026-08-15). Divide the budget instead:
//--- an ensemble chart's UI latency returns to the solo chart's (~4 x 30ms) at the cost of a little
//--- more per-chunk dispatch overhead, which the derived ~29k-weight topology can afford.
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
//--- 2026-08-16: the divided budget alone did NOT restore responsiveness, because all members shared
//--- one custom-event id and CExpertCustom broadcasts events to every filter - each posted event ran
//--- a chunk in ALL N members, N*N chunks per round, and the thread never idled (panel completely
//--- dead, not just sticky). Fixed with per-instance study-event ids (see STUDY_EVENT_ID_BASE); the
//--- divided budget below is what makes the FIXED dispatch behave as the note above intends.
//--- Divided by the ACTIVE trainer count, not the member count: a member waiting at the era barrier
//--- (or deployed/paused) consumes no chunks, so its share is donated to the members still working -
//--- one laggard left gets the full 120ms - while the chart thread's UI headroom stays constant.
const uint TRAIN_TIME_BUDGET_MS = m_ensembleMember ? (uint)(120 / MathMax(EnsembleActiveTrainers(), 1)) : 120;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//---
//--- Never block the calling thread while paused/stopped - just decline this call (or finalize a
//--- run that just got stopped) and let the next scheduled call check again, so Pause/Resume/Stop
//--- and everything else on the control panel stays responsive instead of Sleep()-ing the one
//--- MQL5 thread this chart has.
if(m_trainingPaused && !IsStopped() && !m_trainingStopRequested)
return;
bool stop = IsStopped() || m_trainingStopRequested;
if(stop)
{
if(m_trainRunActive)
FinalizeTrainRun();
if(m_simOosRunActive)
{
delete m_simOosNet;
m_simOosNet = NULL;
m_simOosRunActive = false;
}
return;
}
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//--- ENSEMBLE DEPLOY, approved by the ensemble gate on some member's era end (see
//--- EnsembleEraVerdict). Acted on HERE, at the next call, rather than at the next era end: the
//--- decision is already made, and training one more era would only produce weights that
//--- FinalizeTrainRun immediately discards in favour of the joint checkpoint. Each member restores
//--- its own half of that checkpoint, so the quartet that goes live is the one the vote was
//--- measured on. A member without a snapshot keeps training rather than deploying weights nothing
//--- measured - the verdict refuses to approve in that case, so this is belt and braces.
if(m_ensembleMember && g_ensDeployApproved && !m_trainingComplete &&
m_haveOosCheckpoint && m_checkpointEra == g_ensBestEra)
{
m_trainingComplete = true;
Print(ID + ": ENSEMBLE DEPLOY - restoring this model's weights from the joint checkpoint at era " +
IntegerToString((int)g_ensBestEra) + " and switching to live inference. The combined vote,"
" not this model alone, is what cleared the gate.");
if(m_trainRunActive)
FinalizeTrainRun();
fix: the DB backfill could never run, and HEAD did not compile Four defects in 64c5dd5/1a05e63, found by review + a baseline compile. Goals 1-8 of that session are unchanged; this makes 6 and 8 actually reachable. 1. HEAD DID NOT COMPILE - 6 errors. CControlPanel::Minimize/Maximize were declared `virtual bool ... override`, but CAppDialog declares both as `virtual void` (Controls\Dialog.mqh). errors 265 + 404 on each, plus 151 on `bool ok = CAppDialog::Minimize()`. Return type is void now; there was never a success flag to forward. Verified: 0 errors, 0 warnings. 2. THE BACKFILL COULD NEVER ADVANCE, and neither could the OOS continual simulation (that one has been dead since it was written). Both are armed at the instant convergence is declared, and both advance only from inside Train(), one chunk per call. But ScheduleTrainingIfNeeded's only per-tick ArmStudyEvent site sits in the `else` of a branch taken whenever m_trainingComplete is set and m_trainRunActive is clear - which is exactly the state FinalizeTrainRun() leaves behind one line before they are armed. Train() was never called again, so the walks sat at their start index forever: no "simulation complete" line, and not one row written to the DB this feature exists to fill. Only a manual Resume/Retrain unstuck them. Both flags now keep the model schedulable. 3. IN AI_HYBRID - the mode this ships in - the backfill was never even armed. Ensemble members deploy at Train() ENTRY and return immediately (so no era is wasted), which skips the era-end block the backfill was started from. All four members were a no-op for a second, independent reason. Armed on the ensemble deploy path too, from m_resumeBars/m_resumeOosCutoff. 4. RE-RUNS DUPLICATED ROWS. RegisterSignal inserts unconditionally - no key, no duplicate check - and m_dbBackfillDone is in-memory, so every later attach that retrained to convergence wrote a second full set of rows for the same bars. The ranking would count one bar once per model that ever deployed, weighting superseded opinions as heavily as the live one. A .dbfill marker stamps the deployed era; written only on completion (an interrupted walk redoes itself rather than ranking a partial window) and deleted with the other sidecars on reset-weights. Also: WarmBlocking's timeout was silent, which restored the exact silent pin failure it was added to prevent - it now says so in the journal, and returns true for "no reference pairs to wait for" so the warning stays rare enough to be read. Not addressed, needs a decision: the backfill scores the OOS window with the checkpoint that was SELECTED as best on that same window, then writes those win rates into the table filter weights rank on - the selection set consumed twice, undiscounted, while the deploy gate right next to it applies a family-wise correction for exactly that effect. The rows are also simulated triple-barrier outcomes at today's spread sharing a table with realised fills. The completion log line now states both plainly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 21:25:51 -04:00
//--- Same one-shot pattern-database backfill the solo path arms at its era end, and for the same
//--- reason (see StartPatternDatabaseBackfill): a deployed model has to be RANKED the instant it
//--- goes live, not an hour of real trades later. It has to be armed HERE as well because an
//--- ensemble member never reaches that era-end block - deploy is acted on at Train() ENTRY and
//--- returns immediately (that is the whole point: no wasted era), so line ~3489 is unreachable
//--- for all four members and the feature was a no-op in exactly the mode it ships in.
//--- m_resumeBars/m_resumeOosCutoff are the last era's own window bounds (see where the era loop
//--- stamps them) - the same pair the solo call passes, just read from the members that survive
//--- across chunked calls rather than from the era loop's locals, which are out of scope here.
fix: drop the ranking slice for the calibration band; un-collapse the tiers NOT COMPILED - user compiles. (1) THE RANKING SLICE IS GONE. It reserved 20% of the OOS window so the pattern-DB backfill would read bars the deployed checkpoint was not SELECTED on. That objection stands; carving a new region to answer it did not. The calibration band already has every property the slice was buying: never trained on | never graded by pass 3 (which walks [0, oosCutoff) and so never reaches it) | never seen by the deploy gate | purged by a full label horizon on BOTH sides | and larger besides - 1,684 bars vs the ~970 carved So the backfill now walks [calibLo, calibHi) and pass 3 goes back to grading the entire OOS window, exactly as before any of this. The gate gets its full sample back (~10% of a sigma), the split loses a region, and the failure mode found an hour ago - a reserved region silently blanking ~10 months of chart arrows, because arrows are only drawn on bars pass 3 grades - becomes impossible. One impurity, stated in the completion log rather than hidden: m_dirConfThreshold is FITTED on that band and the walk applies it to decide which bars fired, so coverage there is mildly optimistic. One scalar under a coverage floor, against checkpoint selection over hundreds of eras. This backfill IS the deploy-time warm-up: it runs right after FinalizeTrainRun() restores the deployed weights, so it scores with exactly what is about to trade. (2) EVERY CALL WAS TIER 0, AND IT WAS ARITHMETIC. ConfidenceTier() quartiles [floorConf, 1] where floorConf = 1/3 - the lowest magnitude a 3-way softmax winner can hold. But it was fed CalibratedConfidenceMagnitude(), which multiplies by m_confidenceCalScale, clamped to [0.3, 1.5]. That lower clamp is BELOW 1/3. Whenever calibration bottoms out, t goes negative and MathMax(0, ...) pins every call to tier 0. Which is what the live run does. m_confidenceCalScale is EMA'd toward empiricalAccuracy / avgClaimedConfidence; with the model over-calling Neutral, 3-class agreement sits near 10% against a claimed confidence near 0.9, so the ratio is ~0.11 and clamps to 0.3 every era. Logged: tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0) 828 calls, one bucket - the four tier weights and the entire per-tier pattern-DB ranking reduced to a single number. The backfill was feeding a mechanism that structurally could not rank. Tiering now reads the RAW head magnitude, which genuinely lives on the [1/3, 1] range these bounds were written for. Calibration keeps its real jobs - AIConfidence() for MM sizing and SignedAIConfidence() for the vote are unchanged. STILL OPEN, deliberately not touched here: the calibration TARGET itself. empiricalAccuracy is 3-class agreement, which is the wrong quantity to scale a DIRECTIONAL confidence against - it counts a Neutral class that is 0.19% of labels. The honest target is the win rate on the calls the confidence describes (directional precision), with the claimed-confidence average taken over those same called bars. That needs a new accumulator and it interacts with the Neutral over-calling being fixed elsewhere, so it wants one clean run first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 23:06:11 -04:00
StartPatternDatabaseBackfill(m_resumeBars, m_resumeTotalIter, m_resumeOosCutoff);
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
return;
}
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
//--- ENSEMBLE ERA BARRIER (user request 2026-08-16): members advance era by era TOGETHER, because
//--- the number that matters - the combined-vote OOS score - is only well-defined when every
//--- member's pass 3 describes the same era, and because live trading is the members voting
//--- together, not four models drifting apart in training age. A member ahead of the slowest
//--- still-training member declines the call here; it costs nothing (its chunk budget flows to the
//--- laggards via EnsembleActiveTrainers above) and resumes untouched when the barrier clears.
//--- Deployed, stopped and paused members are exempt from the min (see EnsembleMinTrainingEra), so
//--- nothing deadlocks. Placed AFTER the pause/stop handling: a Stop must still finalize, and a
//--- barred member must still respond to the panel.
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
//--- Stamp this member's era-advance clock BEFORE the barrier test, so a member that cannot finish an
//--- era eventually stops pinning the whole chart (see BarrierEraHeartbeat / ENSEMBLE_BARRIER_STUCK_MS).
BarrierEraHeartbeat();
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
if(EnsembleEraBarrierHolds())
{
//--- deliberate idleness, not a stall - keep the stall watchdog's era clock current and say
//--- what is happening on the member's panel line instead of freezing its last progress text
m_lastEraCompleteTick = GetTickCount();
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
//--- AND SAY SO IN THE JOURNAL. Resetting the watchdog above is right (a held member is idle, not
//--- stalled) but it was the ONLY thing this branch did, so a held member left no record anywhere.
//--- On 2026-08-17 that meant two frozen charts reported through their two BROKEN members and
//--- stayed completely silent about the two healthy ones - the panel said "Waiting at era N" and
//--- the journal, which is what gets read afterwards, said nothing at all. Rate-limited, and it
//--- names the member being waited on so the blocker is identified from one line rather than by
//--- cross-referencing every member's last era.
uint nowTick = GetTickCount();
feat(logs): throttle the settled per-era diagnostics - measured 22MB/9.5h of confirmed-working systems Measured from the journal (2026-08-19): the era deep-dive line (~2KB) plus the excursion verdict, tier re-rank, calibration move, barrier hold and selection-regressed note each printed EVERY era for EVERY member - ~940 eras/member/day - long after the systems they watch were confirmed working. Yesterday's file was 1.3GB (70% of it the news-filter calendar spam the sweep fix already removed). VerboseMode returns as an INPUT (demoted 2026-08-01 for the marketplace; that track is dead since the 2026-08-16 pivot) and gains a second job: false throttles each settled per-era print to eras 0-3 plus every TRAIN_LOG_EVERY_ERAS-th (25 ~= one deep-dive per ~15min per member); true restores the per-era firehose, flippable live. Never throttled: anything that marks a CHANGE - new bests, restores + eta decays, plateau stage transitions, deploy approvals, warnings, errors, the label-cache/adoption one-shots, and the combined-vote gate line (the active system's primary telemetry, still every era). Semantic fixes over blanket gating: - barrier hold now ARMS silently and prints only when the hold outlasts the 2-min report interval - a brief hold every era is the design, the long hold is the watchdog case the line exists for; - the ensemble deploy REFUSAL prints immediately when its reason changes (that is a finding), on cadence when unchanged; - the filtered-view census prints when its RESULT moves (drawn count, or strongest vote by >=2pp) and at least every 10th sweep. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 09:36:15 -04:00
//--- ARM SILENTLY, REPORT ONLY WHEN THE HOLD OUTLASTS THE INTERVAL (2026-08-19). Printing on
//--- entry logged ~950 lines/member/day, because a brief hold at the barrier is the DESIGN -
//--- the fast member waits a few seconds here every era. The line's own text says what it is
//--- for ("if this line keeps repeating..."): the pathological case is a LONG hold. So a
//--- hold becomes a journal line only once it has lasted a full report interval; the healthy
//--- per-era waits never print at all. VerboseMode restores the on-entry print.
bool justHeld = (m_barrierHoldReportTick == 0);
if(justHeld)
m_barrierHoldReportTick = nowTick;
if((VerboseMode && justHeld) ||
nowTick - m_barrierHoldReportTick >= ENSEMBLE_BARRIER_REPORT_MS)
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
{
m_barrierHoldReportTick = nowTick;
long minEra = EnsembleMinTrainingEra();
string blockers = "";
for(int bi = 0; bi < ArraySize(g_warriorEnsemble); bi++)
{
CExpertSignalAIBase *bm = g_warriorEnsemble[bi];
if(CheckPointer(bm) == POINTER_INVALID)
continue;
if(bm.m_trainingComplete || bm.m_trainingStopRequested || bm.m_trainingPaused ||
!bm.m_isInitialized || bm.m_barrierExcluded)
continue;
if(bm.m_eraCount <= minEra)
blockers += (blockers == "" ? "" : ", ") + bm.ID;
}
PrintFormat("%s: HELD AT THE ERA BARRIER - this member is at era %d and the ensemble minimum is"
" %d, so it is idle until [%s] catch up. It is NOT stalled and its weights are"
" untouched. If this line keeps repeating, the member(s) named are the ones to"
" diagnose - after %d minutes without an era they are dropped from the barrier and"
" this member resumes on its own.",
ID, (int)m_eraCount, (int)minEra, blockers == "" ? "(none - resolving)" : blockers,
(int)(ENSEMBLE_BARRIER_STUCK_MS / 60000));
}
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
PublishStatus(StringFormat("Waiting at era %d for slower ensemble members (min era %d) - donating its compute until they catch up",
(int)m_eraCount, (int)EnsembleMinTrainingEra()));
return;
}
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
m_barrierHoldReportTick = 0;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- Evaluation-only continual-learning OOS simulation walk in progress (see StartOosContinualSimulation):
//--- give it exclusive occupancy of this call, same chunked budget as the real era loop below, so a
//--- large OOS window can't freeze the UI in one shot. While it's active no real-training
//--- ResizeBuffers()/RefreshData() runs, so the price/ATR/time buffers it reads stay frozen for its
//--- whole walk - it never has to worry about the label cache's shifting-index invalidation below.
if(m_simOosRunActive)
{
fix: prebuild and era sized different windows; diag: Train() names its branch TWO things, one incident. 1) THE BUG I SHIPPED IN 0c85c54. m_tuneStartTrainBar is declared, initialised to 0, and NEVER ASSIGNED - the assignment existed before the God-class split and the split dropped it, leaving a dead member. Harmless while nothing read it; a real defect the moment 0c85c54 made StartLabelCachePrebuild() reset dtStudied from it. Train() then computed the window as max(StartTrainBar, floor) while the prebuild computed max(0, floor), where StartTrainBar is the non-zero datetime OnChartEventHandler passes through from the "New Bar" event. The two therefore disagreed about `bars`, so EnsureBarCachesCapacity() saw a changed size at era start, wiped the caches, and re-armed a full 38k-bar prebuild - instead of training. Restored the assignment so both sides evaluate the identical expression. 2) THE REASON IT TOOK ALL NIGHT TO FIND. Train() is a state machine with six early-return branches above the era loop and every one of them is silent. Four charts burned a core each for 15 minutes with an empty journal: the pass heartbeats (694b756) proved the era loop was never reached, no prebuild completion line appeared either, and nothing external can see inside a single MQL5 thread - per-thread CPU says "busy", file writes say nothing, and the VPS has no debugger. That is an undiagnosable state, and it is the thing to fix, not just the bug of the day. ReportTrainStall() now names the branch Train() is taking whenever no era has completed for 3 minutes, at most once a minute per signal, with the state that decides the branch: run/prebuild/simOos/resume flags, era, dtStudied, and - for the cache-invalidation branch specifically - BOTH bar counts, since two sizings disagreeing is exactly what re-arms the prebuild forever. Silent on a healthy run: an era completing resets the clock. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:32:08 -04:00
ReportTrainStall("OOS continual-learning simulation walk");
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
AdvanceOosSimulationChunk();
return;
}
feat: implement one-shot pattern-database backfill and enhance accuracy tracking for ensemble models
2026-08-16 21:08:41 -04:00
//--- One-shot pattern-database backfill in progress (see StartPatternDatabaseBackfill) - same
//--- exclusive-occupancy/chunking treatment as the simulation walk above.
if(m_dbBackfillActive)
{
ReportTrainStall("pattern-database backfill walk");
AdvancePatternDatabaseBackfill();
return;
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- Eager label-cache pre-build in progress (see StartLabelCachePrebuild/AdvanceLabelCachePrebuild) -
//--- same exclusive-occupancy/chunking treatment as the OOS simulation walk above, so it can't freeze
//--- the UI on a large study window either. m_trainRunActive stays false for its whole duration, so
//--- once it completes, Train() falls through to the normal !m_trainRunActive setup below and era 0
refactor(ai): extract Layer.mqh and deduplicate AI config - Moves CLayer neuron construction to AI/Impl/Layer.mqh to keep Network.mqh clean - Unifies four previously duplicated architecture initialisation blocks (MLP/CONV/LSTM/HYBRID) into a single shared function - Eliminates risk of behavioural drift where one architecture missed a setter, causing mismatched feature sets or targets
2026-08-01 11:27:28 -04:00
//--- starts from the measured class distribution it just seeded.
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
if(m_labelPrebuildActive)
{
fix: prebuild and era sized different windows; diag: Train() names its branch TWO things, one incident. 1) THE BUG I SHIPPED IN 0c85c54. m_tuneStartTrainBar is declared, initialised to 0, and NEVER ASSIGNED - the assignment existed before the God-class split and the split dropped it, leaving a dead member. Harmless while nothing read it; a real defect the moment 0c85c54 made StartLabelCachePrebuild() reset dtStudied from it. Train() then computed the window as max(StartTrainBar, floor) while the prebuild computed max(0, floor), where StartTrainBar is the non-zero datetime OnChartEventHandler passes through from the "New Bar" event. The two therefore disagreed about `bars`, so EnsureBarCachesCapacity() saw a changed size at era start, wiped the caches, and re-armed a full 38k-bar prebuild - instead of training. Restored the assignment so both sides evaluate the identical expression. 2) THE REASON IT TOOK ALL NIGHT TO FIND. Train() is a state machine with six early-return branches above the era loop and every one of them is silent. Four charts burned a core each for 15 minutes with an empty journal: the pass heartbeats (694b756) proved the era loop was never reached, no prebuild completion line appeared either, and nothing external can see inside a single MQL5 thread - per-thread CPU says "busy", file writes say nothing, and the VPS has no debugger. That is an undiagnosable state, and it is the thing to fix, not just the bug of the day. ReportTrainStall() now names the branch Train() is taking whenever no era has completed for 3 minutes, at most once a minute per signal, with the state that decides the branch: run/prebuild/simOos/resume flags, era, dtStudied, and - for the cache-invalidation branch specifically - BOTH bar counts, since two sizings disagreeing is exactly what re-arms the prebuild forever. Silent on a healthy run: an era completing resets the clock. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:32:08 -04:00
ReportTrainStall("label-cache prebuild scan");
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
AdvanceLabelCachePrebuild();
return;
}
if(!m_trainRunActive)
{
//--- Wait (briefly, bounded, non-blocking across calls) for the terminal to finish syncing this
//--- symbol/period's history from the broker before computing the training window. Bars(symbol,
//--- period) - the hard cap on how many bars the era loop below will ever process - reflects
//--- whatever's synced SO FAR, not necessarily the true total; starting before sync completes
//--- would let that cap (and therefore the "Bar X of Y" progress display) silently grow between
//--- eras as more history trickles in.
if(!SeriesInfoInteger(m_symbol.Name(), PERIOD_CURRENT, SERIES_SYNCHRONIZED))
{
uint syncNowTick = GetTickCount();
if(m_syncWaitStartTick == 0)
m_syncWaitStartTick = syncNowTick;
if(syncNowTick - m_syncWaitStartTick < 5000)
fix: prebuild and era sized different windows; diag: Train() names its branch TWO things, one incident. 1) THE BUG I SHIPPED IN 0c85c54. m_tuneStartTrainBar is declared, initialised to 0, and NEVER ASSIGNED - the assignment existed before the God-class split and the split dropped it, leaving a dead member. Harmless while nothing read it; a real defect the moment 0c85c54 made StartLabelCachePrebuild() reset dtStudied from it. Train() then computed the window as max(StartTrainBar, floor) while the prebuild computed max(0, floor), where StartTrainBar is the non-zero datetime OnChartEventHandler passes through from the "New Bar" event. The two therefore disagreed about `bars`, so EnsureBarCachesCapacity() saw a changed size at era start, wiped the caches, and re-armed a full 38k-bar prebuild - instead of training. Restored the assignment so both sides evaluate the identical expression. 2) THE REASON IT TOOK ALL NIGHT TO FIND. Train() is a state machine with six early-return branches above the era loop and every one of them is silent. Four charts burned a core each for 15 minutes with an empty journal: the pass heartbeats (694b756) proved the era loop was never reached, no prebuild completion line appeared either, and nothing external can see inside a single MQL5 thread - per-thread CPU says "busy", file writes say nothing, and the VPS has no debugger. That is an undiagnosable state, and it is the thing to fix, not just the bug of the day. ReportTrainStall() now names the branch Train() is taking whenever no era has completed for 3 minutes, at most once a minute per signal, with the state that decides the branch: run/prebuild/simOos/resume flags, era, dtStudied, and - for the cache-invalidation branch specifically - BOTH bar counts, since two sizings disagreeing is exactly what re-arms the prebuild forever. Silent on a healthy run: an era completing resets the clock. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:32:08 -04:00
{
ReportTrainStall("waiting for history sync");
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
return; // retry on the next scheduled call instead of blocking here
fix: prebuild and era sized different windows; diag: Train() names its branch TWO things, one incident. 1) THE BUG I SHIPPED IN 0c85c54. m_tuneStartTrainBar is declared, initialised to 0, and NEVER ASSIGNED - the assignment existed before the God-class split and the split dropped it, leaving a dead member. Harmless while nothing read it; a real defect the moment 0c85c54 made StartLabelCachePrebuild() reset dtStudied from it. Train() then computed the window as max(StartTrainBar, floor) while the prebuild computed max(0, floor), where StartTrainBar is the non-zero datetime OnChartEventHandler passes through from the "New Bar" event. The two therefore disagreed about `bars`, so EnsureBarCachesCapacity() saw a changed size at era start, wiped the caches, and re-armed a full 38k-bar prebuild - instead of training. Restored the assignment so both sides evaluate the identical expression. 2) THE REASON IT TOOK ALL NIGHT TO FIND. Train() is a state machine with six early-return branches above the era loop and every one of them is silent. Four charts burned a core each for 15 minutes with an empty journal: the pass heartbeats (694b756) proved the era loop was never reached, no prebuild completion line appeared either, and nothing external can see inside a single MQL5 thread - per-thread CPU says "busy", file writes say nothing, and the VPS has no debugger. That is an undiagnosable state, and it is the thing to fix, not just the bug of the day. ReportTrainStall() now names the branch Train() is taking whenever no era has completed for 3 minutes, at most once a minute per signal, with the state that decides the branch: run/prebuild/simOos/resume flags, era, dtStudied, and - for the cache-invalidation branch specifically - BOTH bar counts, since two sizings disagreeing is exactly what re-arms the prebuild forever. Silent on a healthy run: an era completing resets the clock. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:32:08 -04:00
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
Print(ID + ": WARNING - history for " + m_symbol.Name() + " " + EnumToString(PERIOD_CURRENT) + " did not finish syncing after 5s; training window may still grow as more history arrives");
}
m_syncWaitStartTick = 0;
//--- 3 no-op passes before the era loop ever runs for a fresh start (see m_warmupPassesRemaining's
//--- declaration comment) - each is its own separately-scheduled Train() call (this whole method
//--- just returns, deferring to the next "New Bar"/timer-driven call), giving MT5's history sync
//--- several real, wall-clock-separated chances to settle on top of the 5s soft wait just above,
//--- before training commits to a bar count and starts populating the label cache below.
if(m_warmupPassesRemaining > 0)
{
fix: prebuild and era sized different windows; diag: Train() names its branch TWO things, one incident. 1) THE BUG I SHIPPED IN 0c85c54. m_tuneStartTrainBar is declared, initialised to 0, and NEVER ASSIGNED - the assignment existed before the God-class split and the split dropped it, leaving a dead member. Harmless while nothing read it; a real defect the moment 0c85c54 made StartLabelCachePrebuild() reset dtStudied from it. Train() then computed the window as max(StartTrainBar, floor) while the prebuild computed max(0, floor), where StartTrainBar is the non-zero datetime OnChartEventHandler passes through from the "New Bar" event. The two therefore disagreed about `bars`, so EnsureBarCachesCapacity() saw a changed size at era start, wiped the caches, and re-armed a full 38k-bar prebuild - instead of training. Restored the assignment so both sides evaluate the identical expression. 2) THE REASON IT TOOK ALL NIGHT TO FIND. Train() is a state machine with six early-return branches above the era loop and every one of them is silent. Four charts burned a core each for 15 minutes with an empty journal: the pass heartbeats (694b756) proved the era loop was never reached, no prebuild completion line appeared either, and nothing external can see inside a single MQL5 thread - per-thread CPU says "busy", file writes say nothing, and the VPS has no debugger. That is an undiagnosable state, and it is the thing to fix, not just the bug of the day. ReportTrainStall() now names the branch Train() is taking whenever no era has completed for 3 minutes, at most once a minute per signal, with the state that decides the branch: run/prebuild/simOos/resume flags, era, dtStudied, and - for the cache-invalidation branch specifically - BOTH bar counts, since two sizings disagreeing is exactly what re-arms the prebuild forever. Silent on a healthy run: an era completing resets the clock. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:32:08 -04:00
ReportTrainStall("history-settle warm-up pass");
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_warmupPassesRemaining--;
PrintVerbose(ID + ": warm-up pass " + IntegerToString(3 - m_warmupPassesRemaining) + " of 3 (letting history sync settle before training starts)");
return;
}
fix: a restart no longer loses the measured geometry or the training window Terminal restart, 22:25: all four resumed models sat on empty windows with enum 2:6 barriers. Three interlocking causes, all visible in one log excerpt: 1) THE PRE-SCAN WINDOW WAS SIZED BY THE SAVED WATERMARK. A resumed model's dtStudied sits at its last studied bar, so Bars(dtStudied, now) ~ 0 and the resumed-model MI pre-scan built a zero-bar "complete" label cache - logged as "Buy: 0 | Sell: 0 | Neutral: 0". Train()'s own era start RESETS dtStudied to the training-window rule before computing its window; the pre-scan did not. The rule is now factored into TrainWindowStart() and both use it. The scan also refuses to arm before SERIES_SYNCHRONIZED (it ran in the same second as OnInit), and deployed models keep their watermark - for them it gates inference recency, not a training window. 2) THE HORIZON LATCHED ON AN INDICATOR WARM-UP. ComputeBarrierHorizonBars ran against a ZigZag with 0 calculated legs, fell back, and EnsureBarrierHorizon latched fallback(32) x slMult x tpMult = 384 for the process lifetime. A leg-starved horizon is now PROVISIONAL: re-resolved on the next rebuild, the label cache wiped if it moved (labels from two horizons answer different questions), and the geometry deriver refuses to run from it - a pair derived over a warm-up window would get PINNED. 3) THE DERIVED GEOMETRY WAS NEVER PERSISTED. The .cfg is written at model creation and at weights-reset - both BEFORE era 0 derives - so the measured pair lived only in memory: every restart read back zeros, adopted nothing, fell back to the enum barriers, and the era-0-only gate meant a resumed model could NEVER re-derive. A full day of training on 3.33/1.62 resumed as 2:6. Now: the settled pair is pinned to the .cfg the moment derivation completes (one-shot, atomic write), and the derive gate accepts any model with no pinned pair, not just era 0 - mid-run stability is carried by m_geometryDerived itself, which never allows a second derivation. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 22:40:43 -04:00
//--- ALL available history, floored by MinTrainYear - see TrainWindowStart(). Factored out
//--- (2026-08-09) because StartLabelCachePrebuild needs the SAME rule: the resumed-model
//--- pre-scan used to size its window from the SAVED dtStudied instead, and a model whose
//--- watermark sat at the last studied bar got Bars(dtStudied, now) = 0 - a zero-bar "complete"
//--- label cache, logged as "Buy: 0 | Sell: 0 | Neutral: 0", with everything downstream
//--- (horizon, geometry, the MI report) computed on nothing.
dtStudied = TrainWindowStart(StartTrainBar);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- OOS-based objective + stability tracking: training only "converges" once the objective is
//--- met AND OOS accuracy has held inside a tight band for the last few eras, so a single lucky
//--- era can't get locked in as the final model. The best-scoring era's weights are checkpointed
//--- to an agent-local scratch file (not FILE_COMMON) and restored at the end - this works inside
//--- the tester too, unlike Net.Save()/Load() which are disabled there.
m_oosWindow.Clear();
m_bestOosForecast = -1;
m_bestBalancedOos = -1;
m_bestPassedRecall = false;
fix: a one-sided era can no longer become the best checkpoint Measured on HYBRID, era 29 of the first win-scored run: the model collapsed to always-Buy and was crowned "new best selection score 67.1%". Under win-based scoring that is not a coincidence - the always-call-the-drift-side model IS the chance reference, so it scores exactly chance (P(winLong) ~ 67% on SP500), while every honest two-sided era scores 63-66% because shorts win less often against the drift. Raw score ranking therefore actively prefers the degenerate model, every regression restores back to it, and live NMS collapses its near-constant signal to ~25 trades per era - observed as "hybrid barely trades". bothSidesLive already blocked one-sided eras from DEPLOYING (tradeableOK, 371f8aa), but among not-yet-deployable eras the score alone ranked - the same early phase the coverage credit was added for, failing the same way through a different door. The ranking key is now three lexicographic tiers: deployable > two-sided > score. A one-sided era cannot displace a two-sided best regardless of score - by construction its score is a property of the data's drift, not the model - and a two-sided era displaces a one-sided best no matter how much lower it scores. m_bestBothSidesLive is snapshotted with the checkpoint and reset with the rest of the best-tracking state. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 21:19:44 -04:00
m_bestBothSidesLive = false;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_haveOosCheckpoint = false;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
m_checkpointEra = -1; // the joint-checkpoint era stamp goes with the snapshot it describes
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_oosStable = false;
m_objectiveMet = false;
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
//--- Family-wise deployment gate state, reset with the checkpoint tracking it describes: N counts
//--- the eras THIS run selects a maximum over, so carrying it across runs would test the winner
//--- against a search that never happened.
m_bestDirPrecPct = -1.0;
m_bestChancePrecPct = -1.0;
m_bestDirCalls = 0;
m_deployCandidateEras = 0;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_erasSinceCooldown = 0;
m_eraResumePending = false;
refactor(ai): nine class-imbalance inputs down to two The imbalance section offered nine controls for one job. Audited against the code, five of them did not do what their names said at the shipped defaults: AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns whenever the adjusted loss is on, which is default. OversampleParity DEAD in training - Training.mqh gated the replay loop on !useLogitAdjustedLoss (correctly, citing Buda et al. 2018). Live only in the online-learning path. EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma damper - "replay minority bars through pass-2 oversampling" was a focal-loss switch. ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25. UseStaticPrior An exact duplicate of FreezePriorCalibration - the two were OR'd together in the single place either is read. So they were not five mechanisms fighting; they were one mechanism plus eight knobs that mostly described machinery that no longer ran. That is worse than a real conflict, because the log agreed with the names: the label-cache line printed "reps up to 28x (90% parity) (seeding era 0's class-balance oversampling)" on every run, describing an oversampling pass that had been switched off. It is fixed here too - it cost this session a wrong diagnosis. The one genuine redundancy was focal loss, running at gamma*0.125 alongside the adjusted loss: two corrections on the same axis, the exact stacking failure this file already cited Buda et al. for in two other places, damped by a replay flag whose replay path was itself dead. Removed rather than re-tuned. The plateau ladder is unaffected - its escape is the learning-rate warm restart; the gamma anneal beside it only ever stepped toward zero. WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze: LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted- Loss boolean, since a strength dial where 0 already means off does not need an on/off switch beside it. FreezePriorCalibration unchanged. It is the only one of the six corrections with a consistency guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection already ranks on - so the loss and the deploy decision optimize one thing. The online continual-learning path keeps its own alpha-balanced focal weight, now as constants pinned to the removed inputs' shipped defaults, so its behaviour is unchanged. It legitimately needs its own correction: ApplyLogitAdjustment() only runs inside a training run, so a deployed model that was reloaded carries no logit offsets and would otherwise stream 31:1 data into itself uncorrected. The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a double fed to a %d conversion and had always emitted a literal 0; the |MR: segment is written as the constant its shipped defaults produced. Dropping either would have re-keyed every model and forced a from-scratch retrain of the one topology currently converged and trading. Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable "neutralized by prior correction" diagnostic. Both builds compile 0 errors, 0 warnings. No retrain forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- Plateau ladder starts fresh with this run, so it re-walks the escalation from its own
//--- starting point. (The focal-gamma anneal that used to reset here went with focal loss on
//--- 2026-07-31 - the ladder's real escape is the learning-rate warm restart.)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_erasSinceBestBalanced = 0;
m_plateauStage = 0;
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
m_restartBoostErasLeft = 0;
feat(search): stop on the IN-SAMPLE plateau, and shrink every best-of-K effect before quoting it Points 3 and 4 of the four-point plan. 1. IN-SAMPLE EARLY STOP - and the reason it is worth having is not compute. The plateau ladder stops on the OOS SELECTION score. That is a peek: by the time it fires, every one of those eras has been evaluated out of sample, so all of them sit in the family the deploy gate corrects over (g_ensCandidateEras, Sidak). Training longer therefore does not merely cost time - it RAISES the bar the eventual winner has to clear. The new stop reads the TRAINING error, which the gate never looks at. When the optimiser has stopped improving on data it can see, more eras will not find a better model; they will only enlarge the OOS family. Ending there shrinks the correction, and the shrinkage is legitimate precisely BECAUSE the stopping rule never consulted an out-of-sample number. That distinction is the whole point and it is the one this project has got wrong four times: stop on IS and the family really is smaller; stop on OOS and those eras were searched and still count. Both stops now exist; only this one buys a lower bar. Deliberately more patient than the OOS ladder (IS_ERROR_PATIENCE_MULT = 3x): training error is noisy per era - mini-batch order alone moves it - and ending a run that is still learning costs far more than a few wasted eras. Improvement is RELATIVE (IS_ERROR_IMPROVE_FRAC = 1%), so it does not depend on the loss's absolute scale, and it only acts when a checkpoint exists, since otherwise it would end a run with nothing to deploy. Reset per RUN alongside the ladder, so a resumed run cannot early-stop on its first era against a previous run's best. 2. WINNER'S-CURSE SHRINKAGE ON THE BARRIER-GEOMETRY WINNER. The family-wise permutation gate already establishes that the RANKING is not noise. It says nothing about the SIZE of the winner's effect - and a best-of-K maximum is biased upward by construction, being the largest of K noisy draws. The adoption message quotes that raw maximum and compares it against the incumbent, so the number a reader plans on is the inflated one. The penalty is now measured, not assumed: the same permutation draws that produce the p-value also produce, per draw, the MAXIMUM excess across all candidates under pure noise. The mean of those maxima is exactly what a best-of-K selection is expected to report when there is nothing there. This is the empirical form of the sqrt(2 ln K) x SE penalty the SQX EdgeFinder plugin applies to every maximum it reports (Stats.java:79-88), and it needs no normality assumption because the draws ARE the null distribution. Applied in James-Stein form - effect x max(0, 1 - penalty^2/effect^2) - so a large effect is nearly untouched and a marginal one collapses toward zero. Reported, not gated. The adoption decision still turns on the permutation p-value, which is the right test for "is the ranking real"; the shrunk number is there so the magnitude quoted beside it is one worth planning on. Closes the first of the two EdgeFinder ports identified on 2026-08-12. NOTE on the second EdgeFinder port, deliberately not done here: "let the measurement steer the target" is already true where it matters most - ReportGeometryExpectancyScan ADOPTS the winning barrier geometry under the family-wise gate rather than advising it, and the MI excursion suite publishes a verdict per instrument per config. What is still missing is steering the TRAINING TARGET itself (direction vs excursion) off those verdicts, and that is a design change rather than a surgical one - direction is a closed verdict while excursion SIZE keeps clearing, so the honest version of that change is a target-selection policy, not a flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:29:00 -04:00
//--- Per-RUN like the ladder above, and for the same reason: a resumed run restarts the search, so
//--- carrying a previous run's best training error would let it early-stop on the first era.
m_bestIsError = -1.0;
m_erasSinceBestIsError = 0;
fix(plateau): the IS-error early stop was inert for every ensemble member 15 hours of training, and the stop that exists to END a run announced itself 1,299 consecutive times without ending anything: SP500 ConvLSTM IN-SAMPLE ERROR PLATEAU - not improved in 1297 / 1298 / 1299 eras (best 0.2689, now 0.3269) ... era 1396, 1397, 1398 SP500 LSTM 536 eras SP500 CONV 442 eras SP500 PAI 150 eras XAUUSD HYB 478 eras XAUUSD LSTM 296 eras XAUUSD CONV 366 eras CAUSE: it wrote its decision into m_plateauStage, and EnsembleEraVerdict mirrors the shared ladder onto every member - `mm.m_plateauStage = g_ensPlateauStage` - on EVERY era, purely so each member's status line shows the collective stage. A display mirror was silently overwriting a decision, so the stop re-armed and re-fired the next era, forever. This is the worst possible direction for this particular bug. Every one of those 1,299 eras was scored out of sample and joined the family the deploy gate corrects over (Sidak, g_ensCandidateEras). The stop's entire purpose is to make that family SMALLER; instead the run spent fifteen hours raising its own bar. - m_isErrorPlateaued: a one-way per-member latch, cleared only by a fresh run. Nothing in the ladder may reset it. The stop condition and the two solo deploy conditions read the latch, not the mirrored stage. - The orchestrator combines: EnsembleEraVerdict requires UNANIMITY across participating members (same participation test the era barrier uses, so an excluded or finished member cannot veto). One member still learning can still move the combined vote, and the vote is what the gate certifies. - Fed in as `dueStage = PLATEAU_STAGE_DEPLOY`, NOT written to g_ensPlateauStage. The block that actually ends the run sits under `dueStage > g_ensPlateauStage`, so assigning the stage directly makes that test false and the deploy never happens - the same inert-write shape as the bug being fixed. Caught before committing; raising dueStage carries it through the ladder's own path (warm restarts skipped, family-wise vote test, measurement screen, joint checkpoint) unchanged. - g_ensIsPlateauAnnounced: announce once per run, not once per era. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 13:09:26 -04:00
m_isErrorPlateaued = false;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//--- ENSEMBLE: the shared gate state is per-RUN for the same reason the per-member state above is -
//--- N must count the eras THIS run's maximum was taken over, so carrying it across runs would
//--- test the winner against a search that never happened.
//--- Reset by the FIRST member to open the run, identified as "no sibling has a run active yet".
//--- Members open their runs within milliseconds of each other but not simultaneously, and a
//--- late starter must not wipe state the ensemble is already accumulating - this test is exact
//--- either way: the first opener sees no active sibling, every later one does, and a member
//--- reopening mid-flight (its own run finalized while the others train on) correctly declines.
bool ensembleRunAlreadyOpen = false;
if(m_ensembleMember)
for(int i = 0; i < ArraySize(g_warriorEnsemble); i++)
{
CExpertSignalAIBase *mm = g_warriorEnsemble[i];
if(CheckPointer(mm) != POINTER_INVALID && mm != GetPointer(this) && mm.m_trainRunActive)
{
ensembleRunAlreadyOpen = true;
break;
}
}
if(m_ensembleMember && !ensembleRunAlreadyOpen)
{
g_ensLastVerdictEra = -1;
g_ensBestScore = -1.0;
g_ensBestTradeable = false;
g_ensBestTwoSided = false;
g_ensBestPrecPct = -1.0;
g_ensBestChancePct = -1.0;
g_ensBestCalls = 0;
g_ensBestEra = -1;
g_ensCandidateEras = 0;
g_ensErasSinceBest = 0;
g_ensPlateauStage = 0;
fix(plateau): the IS-error early stop was inert for every ensemble member 15 hours of training, and the stop that exists to END a run announced itself 1,299 consecutive times without ending anything: SP500 ConvLSTM IN-SAMPLE ERROR PLATEAU - not improved in 1297 / 1298 / 1299 eras (best 0.2689, now 0.3269) ... era 1396, 1397, 1398 SP500 LSTM 536 eras SP500 CONV 442 eras SP500 PAI 150 eras XAUUSD HYB 478 eras XAUUSD LSTM 296 eras XAUUSD CONV 366 eras CAUSE: it wrote its decision into m_plateauStage, and EnsembleEraVerdict mirrors the shared ladder onto every member - `mm.m_plateauStage = g_ensPlateauStage` - on EVERY era, purely so each member's status line shows the collective stage. A display mirror was silently overwriting a decision, so the stop re-armed and re-fired the next era, forever. This is the worst possible direction for this particular bug. Every one of those 1,299 eras was scored out of sample and joined the family the deploy gate corrects over (Sidak, g_ensCandidateEras). The stop's entire purpose is to make that family SMALLER; instead the run spent fifteen hours raising its own bar. - m_isErrorPlateaued: a one-way per-member latch, cleared only by a fresh run. Nothing in the ladder may reset it. The stop condition and the two solo deploy conditions read the latch, not the mirrored stage. - The orchestrator combines: EnsembleEraVerdict requires UNANIMITY across participating members (same participation test the era barrier uses, so an excluded or finished member cannot veto). One member still learning can still move the combined vote, and the vote is what the gate certifies. - Fed in as `dueStage = PLATEAU_STAGE_DEPLOY`, NOT written to g_ensPlateauStage. The block that actually ends the run sits under `dueStage > g_ensPlateauStage`, so assigning the stage directly makes that test false and the deploy never happens - the same inert-write shape as the bug being fixed. Caught before committing; raising dueStage carries it through the ladder's own path (warm restarts skipped, family-wise vote test, measurement screen, joint checkpoint) unchanged. - g_ensIsPlateauAnnounced: announce once per run, not once per era. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 13:09:26 -04:00
g_ensIsPlateauAnnounced = false;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
g_ensDeployApproved = false;
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- One-time eager pre-scan for a fresh start (see m_labelCachePrebuilt's declaration comment) -
//--- kick it off and defer era 0 until it's done, so era 0 can start with a real class-balance
//--- oversampling ratio instead of the reps=1 fallback. Routed via the m_labelPrebuildActive gate
//--- above on every subsequent call until it completes.
if(!m_labelCachePrebuilt)
{
fix: prebuild and era sized different windows; diag: Train() names its branch TWO things, one incident. 1) THE BUG I SHIPPED IN 0c85c54. m_tuneStartTrainBar is declared, initialised to 0, and NEVER ASSIGNED - the assignment existed before the God-class split and the split dropped it, leaving a dead member. Harmless while nothing read it; a real defect the moment 0c85c54 made StartLabelCachePrebuild() reset dtStudied from it. Train() then computed the window as max(StartTrainBar, floor) while the prebuild computed max(0, floor), where StartTrainBar is the non-zero datetime OnChartEventHandler passes through from the "New Bar" event. The two therefore disagreed about `bars`, so EnsureBarCachesCapacity() saw a changed size at era start, wiped the caches, and re-armed a full 38k-bar prebuild - instead of training. Restored the assignment so both sides evaluate the identical expression. 2) THE REASON IT TOOK ALL NIGHT TO FIND. Train() is a state machine with six early-return branches above the era loop and every one of them is silent. Four charts burned a core each for 15 minutes with an empty journal: the pass heartbeats (694b756) proved the era loop was never reached, no prebuild completion line appeared either, and nothing external can see inside a single MQL5 thread - per-thread CPU says "busy", file writes say nothing, and the VPS has no debugger. That is an undiagnosable state, and it is the thing to fix, not just the bug of the day. ReportTrainStall() now names the branch Train() is taking whenever no era has completed for 3 minutes, at most once a minute per signal, with the state that decides the branch: run/prebuild/simOos/resume flags, era, dtStudied, and - for the cache-invalidation branch specifically - BOTH bar counts, since two sizings disagreeing is exactly what re-arms the prebuild forever. Silent on a healthy run: an era completing resets the clock. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:32:08 -04:00
ReportTrainStall("arming the first label-cache prebuild");
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
StartLabelCachePrebuild();
return;
}
m_trainRunActive = true;
}
int bars, totalIter, oosCutoff, i;
bool add_loop;
if(!m_eraResumePending)
{
fix(resume): model reload stalled training - three hardenings on the resume path A resumed META model hot-looped pass 1 (0->100% scan oscillation, silent for 3 minutes until the stall reporter fired) because EVERY window failed at the first AD/Wyckoff feature: the init-time param adoption called ReInitADIndicators unconditionally, destroying five freshly-calculating indicator instances to recreate them with BYTE-IDENTICAL params (verified by parsing the .nnw header - the MI tuner had kept the configured settings), at process start, on a box with 1 GB free of 31. The replacements sat cold for 6+ minutes while full-history resweeps starved the indicator threads harder. - AdoptIndicatorParams: installs a loaded param set into the tuner and rebuilds handles ONLY when the set actually differs from what the live indicators run. Both call sites (resume init + panel reload) use it. - Resumed models get the same 3 warm-up passes as fresh ones. The skip was the shared root cause of the cold-ATR (ba13eef), cold-AD (2026-08-11) and this incident - custom indicators recompute from scratch every process start regardless of what the .nnw proves. - Cold-sweep backoff: a pass-1 sweep in which every window failed on a TRANSIENT cause arms a 5s era-start pause instead of an immediate full-history resweep, so the retry loop stops consuming the CPU/memory the warming indicators need. The stall reporter names the backoff branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 10:23:11 -04:00
//--- COLD-INDICATOR BACKOFF (2026-08-13). When the previous era was discarded because EVERY
//--- window failed on a TRANSIENT cause (an async indicator still calculating - see
//--- ADIndicatorCold/the cold-ATR guard), restarting the sweep immediately is worse than
//--- useless: a full-history pass 1 hammers the CPU and memory the indicator threads need to
//--- finish warming, which on a memory-starved box turns "cold for a second" into "cold
//--- forever" (observed 2026-08-13: a resumed META model resweeping 54k bars back-to-back for
//--- 6+ minutes, indicators never warming, panel oscillating 0->100%). Give them a few quiet
//--- seconds instead; the stall reporter stays the loud diagnosis if it persists.
if(m_coldSweepTick != 0)
{
if(GetTickCount() - m_coldSweepTick < 5000)
{
ReportTrainStall("cold-indicator backoff (all windows failed on a transient cause)");
return;
}
m_coldSweepTick = 0;
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
int barsNow = (int)MathMin(Bars(m_symbol.Name(), PERIOD_CURRENT, dtStudied, TimeCurrent()) + m_historyBars, Bars(m_symbol.Name(), PERIOD_CURRENT));
feat(depth): prime -> settle -> sweep, and name which handle is short "Max bars in chart" is set to Unlimited, so the static-terminal-limit reading in 1dda479 was wrong. Two other candidate causes are falsified too: the price series is fully downloaded and flat (USDJPY 50,162 -> 50,163 over 80 minutes, i.e. one new H4 bar), and the handles have been stable since 13:07 with zero windows for the 17 minutes after, so it is not download-in-progress and not handle churn. The MA period tops out at 200 (ADIndicatorTuner MA_PERIOD_PRESETS) against ~50k bars, so it is not indicator cost either. What IS verified stays verified: CopyBuffer past the calculated depth fails outright rather than short-reading, so the buffer holds nothing and every index reads EMPTY_VALUE; m_MA is the only CiCustom whose block REJECTS on that (m_ADZigZag neutral-fills, RSI/MACD/Ichimoku/ATR are built-ins); the wall is therefore feature 25 of every bar, exactly as the "24 of 832" stall lines said. And it is depth-correlated: 16k-bar charts train, 34k/50k get zero windows forever. So the WHY is still open, and this fix does not depend on it. Per the user's protocol: the request itself is the primer, so prime at full depth, then poll TunableBarsCalculated() every 3s and hold the sweep until it stops changing (3 steady probes), then use whatever it settled at. Bounded at 10 min, and a give-up is logged as a give-up so an abandoned depth is never mistaken for a settled one. This supersedes 1dda479's clamp on the two training paths, which snapshotted a value that may still have been climbing; the clamp remains for the paths that cannot wait (inference/online/rescan/export, see 7e63a8b). The load-bearing part is what does NOT happen while waiting: no sweep. A 50k-bar feature scan starves the indicator threads the request just woke, which is how the failure sustained itself for 40 minutes at a time - discard era, re-sweep, discard, which is the 0->100% oscillation on the panel. Also adds IndicatorDepthReport(): per-handle BarsCalculated() on the priming, cap and stall lines. The logs proved WHICH FEATURE died but never WHICH HANDLE was short, so the cause had to be inferred - and was guessed wrong twice. The next occurrence reads it off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:44:27 -04:00
//--- PRIME, THEN SETTLE, THEN SWEEP.
fix(train): clamp the sweep to indicator-servable depth - the scan wall was CopyBuffer, not a cold indicator Symptom: on a 3-chart run with contention ruled out (SP500 sitting at era 2552), USDJPY and XAUUSD produced 0 usable windows out of 50,163 and 33,966 - forever, re-sweeping on every discard, which is the panel oscillating 0->100%. Bars() is the PRICE series depth. A CUSTOM indicator's is not: MT5 calculates it in its own context bounded by "Max bars in chart" (TERMINAL_MAXBARS), and CopyBuffer past that limit does not short-read, it FAILS - so CDoubleBuffer keeps nothing and EVERY index answers EMPTY_VALUE. ADMovingAverage is the only custom indicator whose feature block REJECTS on EMPTY_VALUE (ADZigZag, also CiCustom, neutral-fills; RSI/MACD/Ichimoku/ATR are built-ins served at any depth), so the sweep died on feature 25 of every bar while the 24 price features under it were fine. That is exactly the "window had 24 of 832 values" the stall report named. Perfectly depth-correlated, measured 2026-08-17: SP500 16,234 bars -> era 2552 XAUUSD 33,982 -> 0 windows XTIUSD 16,611 bars -> era 71 USDJPY 50,179 -> 0 windows This RETIRES the 2026-08-17 cold-indicator reading of the same stall. f0cf659 was right that the rejection must be transient and that the dead backoff had to arm - the branch did change to 'cold-indicator backoff' - but waiting cannot fix a depth the terminal will never grant. So Train() now clamps to TunableBarsCalculated() (which existed and was only ever used for a tuner printout) and trains on the history that IS available, naming TERMINAL_MAXBARS in the log so the cause is readable next time. m_coldSweepTick still owns the genuinely transient case: that reads back as -1, not a positive short count. Recomputed per era, so the clamp lifts by itself if the setting is raised. Also fixes a real off-by-one it was hiding: the MA block reads GetData(idx) AND GetData(idx + 1) for its bar-over-bar change, but ResizeBuffers sized m_MA to barIndex exactly - so the deepest bar of every sweep read one past the end and was rejected as cold. Same shape as the +ichiKijun the Ichimoku/close pair already has. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:27:14 -04:00
//---
feat(depth): prime -> settle -> sweep, and name which handle is short "Max bars in chart" is set to Unlimited, so the static-terminal-limit reading in 1dda479 was wrong. Two other candidate causes are falsified too: the price series is fully downloaded and flat (USDJPY 50,162 -> 50,163 over 80 minutes, i.e. one new H4 bar), and the handles have been stable since 13:07 with zero windows for the 17 minutes after, so it is not download-in-progress and not handle churn. The MA period tops out at 200 (ADIndicatorTuner MA_PERIOD_PRESETS) against ~50k bars, so it is not indicator cost either. What IS verified stays verified: CopyBuffer past the calculated depth fails outright rather than short-reading, so the buffer holds nothing and every index reads EMPTY_VALUE; m_MA is the only CiCustom whose block REJECTS on that (m_ADZigZag neutral-fills, RSI/MACD/Ichimoku/ATR are built-ins); the wall is therefore feature 25 of every bar, exactly as the "24 of 832" stall lines said. And it is depth-correlated: 16k-bar charts train, 34k/50k get zero windows forever. So the WHY is still open, and this fix does not depend on it. Per the user's protocol: the request itself is the primer, so prime at full depth, then poll TunableBarsCalculated() every 3s and hold the sweep until it stops changing (3 steady probes), then use whatever it settled at. Bounded at 10 min, and a give-up is logged as a give-up so an abandoned depth is never mistaken for a settled one. This supersedes 1dda479's clamp on the two training paths, which snapshotted a value that may still have been climbing; the clamp remains for the paths that cannot wait (inference/online/rescan/export, see 7e63a8b). The load-bearing part is what does NOT happen while waiting: no sweep. A 50k-bar feature scan starves the indicator threads the request just woke, which is how the failure sustained itself for 40 minutes at a time - discard era, re-sweep, discard, which is the 0->100% oscillation on the panel. Also adds IndicatorDepthReport(): per-handle BarsCalculated() on the priming, cap and stall lines. The logs proved WHICH FEATURE died but never WHICH HANDLE was short, so the cause had to be inferred - and was guessed wrong twice. The next occurrence reads it off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:44:27 -04:00
//--- This ResizeBuffers/RefreshData pair is the PRIMER: the CopyBuffer it issues at full depth is
//--- what asks the terminal to calculate that far, and asking is the only thing that starts it.
//--- What must NOT follow immediately is the sweep - a 50k-bar feature scan starves the very
//--- indicator threads the request just woke, which is how the 2026-08-17 failure sustained itself
//--- for 40 minutes at a stretch (discard era -> re-sweep -> discard, the panel's 0->100% loop).
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
if(!ResizeBuffers(barsNow) || !RefreshData())
{
fix: training could only advance one 120ms chunk per bar ScheduleTrainingIfNeeded() armed the next Train() call only when dtStudied < lastBarDate. That watermark test is right for a CONVERGED model - one inference refresh per new bar - and wrong for a training run, because Train() is chunked: it does ~120ms of work and yields, needing thousands of calls to finish one era, and every one of those calls has to be armed from there. dtStudied is two incompatible things. Train() sets it to the training WINDOW START (~2008); FinalizeTrainRun() sets it to the last bar SCANNED (~now). So the moment any run finalized, the scheduler went silent until the next candle closed. On H1 that is one chunk per hour. The symptom was indistinguishable from a hang: no era lines, no heartbeats, not one of the six instrumented stall branches - because Train() was not being CALLED. The TRAIN STALL line that caught it reported runActive=Y only because m_trainRunActive had been set microseconds earlier in that same call, and eraResume=N proved no era was in flight. Two log bursts, 28 minutes apart, exactly one H1 bar. Before 0c85c54 this was survivable rather than correct: the saved watermark left almost no bars eligible per era, so eras were nearly free and one call per bar still looked like progress. An unconverged model is now always pending. Pause/stop are handled by m_trainingPaused/m_trainingStopRequested, which Train() checks itself. Also: the one Train() exit that tears down the whole run on a buffer failure was completely silent - it now says so. And the build tag moves to train-dispatch-v2; it had not moved since ce52654, which is why the running binary could not be identified from its own log. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 10:06:49 -04:00
PrintFormat("%s: era start ABORTED - price/indicator buffers would not prepare for %d bars"
feat(depth): prime -> settle -> sweep, and name which handle is short "Max bars in chart" is set to Unlimited, so the static-terminal-limit reading in 1dda479 was wrong. Two other candidate causes are falsified too: the price series is fully downloaded and flat (USDJPY 50,162 -> 50,163 over 80 minutes, i.e. one new H4 bar), and the handles have been stable since 13:07 with zero windows for the 17 minutes after, so it is not download-in-progress and not handle churn. The MA period tops out at 200 (ADIndicatorTuner MA_PERIOD_PRESETS) against ~50k bars, so it is not indicator cost either. What IS verified stays verified: CopyBuffer past the calculated depth fails outright rather than short-reading, so the buffer holds nothing and every index reads EMPTY_VALUE; m_MA is the only CiCustom whose block REJECTS on that (m_ADZigZag neutral-fills, RSI/MACD/Ichimoku/ATR are built-ins); the wall is therefore feature 25 of every bar, exactly as the "24 of 832" stall lines said. And it is depth-correlated: 16k-bar charts train, 34k/50k get zero windows forever. So the WHY is still open, and this fix does not depend on it. Per the user's protocol: the request itself is the primer, so prime at full depth, then poll TunableBarsCalculated() every 3s and hold the sweep until it stops changing (3 steady probes), then use whatever it settled at. Bounded at 10 min, and a give-up is logged as a give-up so an abandoned depth is never mistaken for a settled one. This supersedes 1dda479's clamp on the two training paths, which snapshotted a value that may still have been climbing; the clamp remains for the paths that cannot wait (inference/online/rescan/export, see 7e63a8b). The load-bearing part is what does NOT happen while waiting: no sweep. A 50k-bar feature scan starves the indicator threads the request just woke, which is how the failure sustained itself for 40 minutes at a time - discard era, re-sweep, discard, which is the 0->100% oscillation on the panel. Also adds IndicatorDepthReport(): per-handle BarsCalculated() on the priming, cap and stall lines. The logs proved WHICH FEATURE died but never WHICH HANDLE was short, so the cause had to be inferred - and was guessed wrong twice. The next occurrence reads it off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:44:27 -04:00
" (priming ResizeBuffers/RefreshData failed); ending this training run, it re-arms"
" on the next scheduled call", ID, barsNow);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
FinalizeTrainRun();
return;
}
feat(depth): prime -> settle -> sweep, and name which handle is short "Max bars in chart" is set to Unlimited, so the static-terminal-limit reading in 1dda479 was wrong. Two other candidate causes are falsified too: the price series is fully downloaded and flat (USDJPY 50,162 -> 50,163 over 80 minutes, i.e. one new H4 bar), and the handles have been stable since 13:07 with zero windows for the 17 minutes after, so it is not download-in-progress and not handle churn. The MA period tops out at 200 (ADIndicatorTuner MA_PERIOD_PRESETS) against ~50k bars, so it is not indicator cost either. What IS verified stays verified: CopyBuffer past the calculated depth fails outright rather than short-reading, so the buffer holds nothing and every index reads EMPTY_VALUE; m_MA is the only CiCustom whose block REJECTS on that (m_ADZigZag neutral-fills, RSI/MACD/Ichimoku/ATR are built-ins); the wall is therefore feature 25 of every bar, exactly as the "24 of 832" stall lines said. And it is depth-correlated: 16k-bar charts train, 34k/50k get zero windows forever. So the WHY is still open, and this fix does not depend on it. Per the user's protocol: the request itself is the primer, so prime at full depth, then poll TunableBarsCalculated() every 3s and hold the sweep until it stops changing (3 steady probes), then use whatever it settled at. Bounded at 10 min, and a give-up is logged as a give-up so an abandoned depth is never mistaken for a settled one. This supersedes 1dda479's clamp on the two training paths, which snapshotted a value that may still have been climbing; the clamp remains for the paths that cannot wait (inference/online/rescan/export, see 7e63a8b). The load-bearing part is what does NOT happen while waiting: no sweep. A 50k-bar feature scan starves the indicator threads the request just woke, which is how the failure sustained itself for 40 minutes at a time - discard era, re-sweep, discard, which is the 0->100% oscillation on the panel. Also adds IndicatorDepthReport(): per-handle BarsCalculated() on the priming, cap and stall lines. The logs proved WHICH FEATURE died but never WHICH HANDLE was short, so the cause had to be inferred - and was guessed wrong twice. The next occurrence reads it off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:44:27 -04:00
//--- Now wait out the depth rather than snapshotting it. Returns 0 while the count is still moving.
int settled = SettledBars(barsNow, "training sweep");
if(settled <= 0)
{
ReportTrainStall("priming indicator history (holding the sweep until the calculated depth"
" stops changing)");
return;
}
//--- The floor is the one piece of policy that stays here: below TRAIN_MIN_CLAMPED_BARS a settled
//--- depth is too thin to train anything worth measuring, so the run holds and the stall reporter
//--- stays the loud diagnosis rather than producing a meaningless era.
if(settled < barsNow)
{
if(settled < TRAIN_MIN_CLAMPED_BARS)
{
ReportTrainStall(StringFormat("indicator depth settled at %d bars, below the %d-bar floor"
" for a trainable era", settled, TRAIN_MIN_CLAMPED_BARS));
return;
}
barsNow = settled;
//--- Re-prepare at the clamped depth, and ONLY when it actually changed: the primer above
//--- already left every buffer refreshed at the full depth, so an unconditional second pass
//--- would be a wasted CopyBuffer over every buffer, every era, on the charts that need none.
if(!ResizeBuffers(barsNow) || !RefreshData())
{
//--- The ONLY exit from Train() that tears down the whole run, and it used to be completely
//--- silent - a transient buffer/history hiccup ended the run, FinalizeTrainRun() pushed
//--- dtStudied to the last scanned bar, and the next era simply never started. Indistinguishable
//--- from a hang while it was quiet, so it says so (2026-08-10).
PrintFormat("%s: era start ABORTED - price/indicator buffers would not prepare for the"
" settled depth of %d bars (ResizeBuffers/RefreshData failed); ending this"
" training run, it re-arms on the next scheduled call", ID, barsNow);
FinalizeTrainRun();
return;
}
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
bars = barsNow;
fix(signals): revive a dead MA model, and demote Sanyaku from state to event Two defects surfaced by research/test_classic.py, both verified fixed by re-running the transcription against 178k bars of EURUSD H1. CSignalMA model 1 could never fire. For any recursive average - and MA_TYPE_EMA is the shipped default - MA(i) = a*Close(i) + (1-a)*MA(i+1), so DiffMA(i) = a * (Close(i) - MA(i+1)) DiffCloseMA(i) = (1-a) * (Close(i) - MA(i+1)) are positive multiples of one quantity and always share a sign. Model 1 asks for a close BELOW a RISING average, which is precisely the combination that identity forbids: 0.000% of bars, either direction, any symbol. The MQL5 standard library this was ported from defaults to MODE_SMA, where the two are merely correlated - the bug arrived with the EMA default, not with the port. Reading the slope one bar back (DiffMAPrev) breaks the tie for every MA type while keeping the model's stated meaning. Now fires on 7.92% of bars. CSignalIchimoku model 11 fired on 27% of bars at weight 100. Sanyaku is three standing STATES conjoined with no transition term, so it held across long stretches - and being last in the if-chain at the top weight, the module's highest-conviction reading was also its most common one, overwriting all eight event models below it on a quarter of all bars. The old comment rejected an event form because "demanding all three flip on the same bar would fire almost never" - true, but that is not the alternative. Kouten is the TURN: the ALIGNMENT transitions, and only one role need change for it to. Testing !Sanyaku(idx+1) fires once per aligned stretch. Now 2.17%, in line with Kumo breakout (2.4%) and the strong TK cross (1.1%). DataReady() extended one bar deeper to cover the lookback. Neither pattern showed edge before or after; this is about the models meaning what they say and the vote not being dominated by a constant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:14:34 -04:00
//--- Cross-asset panel is indexed against exactly this bar grid, so it is (re)built wherever
//--- the grid is - never per bar. Non-fatal on failure; see BuildCrossAssetPanel().
BuildCrossAssetPanel(barsNow);
feat(ai): spread as a volatility-regime feature, and fix a stale-index cache in both new blocks Adds spread/ATR and the spread change ratio as network inputs (EnableSpreadFeature, default on). Spread is the one microstructure channel that is both FX-available and genuinely historical in the Strategy Tester - "during testing, the spread is not modeled but is taken from historical data" - so unlike swap, signed tick flow or depth of market it is something a backtest can honestly validate. What it encodes, stated precisely because the raw measurement overstates it. research/test_spread.py found spr/atr the strongest single feature in this codebase, on 5 of 8 instrument/geometry cells at 2-4x any volume feature. But the barrier LABEL charges the spread inside its own barriers, so a wide-spread bar is mechanically likelier to resolve as a loss and the feature would partly be predicting its own cost model. Relabelling at zero cost and re-measuring the identical feature showed 20-40% of it WAS that tautology and the majority was not (XAUUSD retained 97%). What survives is a volatility-regime reading: spread is near-fixed while ATR is not, so the ratio runs high exactly when realised volatility is below its own ATR estimate, which genuinely predicts whether ATR-scaled barriers get reached. It is UNSIGNED - Neutral-vs-directional only, never a side. Also fixes a stale-index bug I introduced with the cross-asset panel and had just repeated in the spread series. Both cached on length alone: if(m_crossAsset.Bars() >= bars) return true; MQL5 series indices are relative to NOW, so one new closed candle shifts every index by one. Keyed only on length, the panel keeps serving its index 0 as a bar that is no longer the newest, and every cross-asset value is read one bar out of step with the price features sitting beside it in the same vector - silently, with no error and no shape change. This is the same class of defect as the dtStudied watermark behind the zero-direction backtests. Both now carry a datetime anchor on m_Time.GetData(0), the same invalidation key the label/feature bar caches already use. And a performance fix that fell out of it: with correct invalidation the panel rebuilds on every new bar, and RefreshConvergedSignal runs per bar - which in the tester would mean one full multi-symbol resample per simulated bar at training depth. Inference only reads bars 0..m_historyBars-1 plus the panel's own slow window, so it now requests exactly that. The cache check is >=, so a deeper panel left from training still satisfies it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:42:40 -04:00
EnsureSpreadSeries(barsNow);
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- Meta target: resolve the candidate corpus onto THIS era's bar grid before pass 1 walks it
//--- (series indices shift on every closed bar, so the resolution is per-era, like the caches).
//--- No candidates is not a trainable state - end the run loudly instead of scanning for nothing.
if(IsMetaTarget() && !MetaPrepareEra(barsNow))
{
PrintFormat("%s: era start ABORTED - no usable meta candidates on this chart (see the"
" MetaCorpus lines above for the corpus/offset diagnostics); ending this training"
" run, it re-arms on the next scheduled call", ID);
FinalizeTrainRun();
return;
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
add_loop = false;
//--- Label/feature cache invalidation: MQL5 timeseries indices are always relative to "now"
//--- (index 0 = current bar), so every new closed candle shifts every older bar's index - a
//--- cache keyed by index would silently misalign the moment that happens. See
//--- EnsureBarCachesCapacity() for why `bars` + m_Time.GetData(0) are the correct/sufficient
//--- invalidation keys.
//--- When a wipe happens MID-RUN (a new candle closed while training was still going - e.g. the
//--- market reopening after the weekend), the label cache comes back empty and the lazy per-bar
//--- fallback (ComputeLabelForBar) labels everything Neutral by design (recent pivots are
//--- unconfirmable) - so continuing on a wiped cache silently turns the REST OF THE RUN into
//--- training AND scoring against an all-Neutral world. Observed 2026-07-19: eras 18-20 started
//--- right after the Sunday session open - IS error collapsed 0.44->0.22, OOS "accuracy" soared
//--- to 84.9% with Buy/Sell recall n/a and era time halved, the convergence machinery happily
//--- rewarding all-Neutral predictions on 100%-Neutral relabeled truth. Re-arm the same chunked
//--- eager prebuild that seeded era 0 and defer this era until it completes; its completion
refactor(ai): extract Layer.mqh and deduplicate AI config - Moves CLayer neuron construction to AI/Impl/Layer.mqh to keep Network.mqh clean - Unifies four previously duplicated architecture initialisation blocks (MLP/CONV/LSTM/HYBRID) into a single shared function - Eliminates risk of behavioural drift where one architecture missed a setter, causing mismatched feature sets or targets
2026-08-01 11:27:28 -04:00
//--- re-seeds the class tallies (m_prebuildSeedPending) so the next era's priors reflect the
//--- freshly relabeled window.
fix(diag): the cache-invalidation stall message could never name the cause SP500 and XAUUSD LSTM wedged at era 1 from 19:43 to 21:00+ (77 min) while their three siblings passed era 200 - the 12-minute barrier exclusion correctly kept the charts alive, so the ensembles ran three-handed. Both printed: cache invalidated at era start (era sized 16236 bars, cache holds 16236) Equal numbers, which reads as "so it wasn't the size". That inference is not available: EnsureBarCachesCapacity assigns BOTH invalidation keys (m_labelCacheBars = bars, m_labelCacheAnchorTime = m_Time.GetData(0)) before it returns true, so a message built afterwards reports the values it just overwrote. The two counts are equal BY CONSTRUCTION and ReportTrainStall's anchor= field is always the live one. The line whose stated job is to name which key tripped was structurally incapable of naming it. Capture bars/anchor BEFORE the call and say which one moved: "SIZE CHANGED 16236 -> 16240" or "size unchanged", and "ANCHOR MOVED 2026.08.18 00:00 -> 04:00" or "anchor unchanged". Not guessing at the cause. An anchor moving every era with the size steady is a new candle each pass or a Time buffer that is not being refreshed; a moving size is the era/prebuild disagreement the branch was written for. The next occurrence will say which, instead of costing another session. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 21:57:12 -04:00
//--- CAPTURED BEFORE THE CALL, and that is the whole point. EnsureBarCachesCapacity assigns BOTH
//--- invalidation keys (m_labelCacheBars = bars, m_labelCacheAnchorTime = m_Time.GetData(0)) before
//--- it returns true, so a message built afterwards reads the values it just overwrote: the two bar
//--- counts are equal BY CONSTRUCTION and the anchor is always the live one. The line that exists
//--- to name which key tripped could therefore never name it, and it printed
//--- "era sized 16236 bars, cache holds 16236" for two members wedged for 77 minutes (SP500 and
//--- XAUUSD LSTM, 2026-08-17 19:43 -> 21:00, stuck at era 1 while their siblings passed era 200).
//--- Equal numbers were read as "not the size then", which is not something that message was ever
//--- able to establish.
int barsBefore = m_labelCacheBars;
datetime anchorBefore = m_labelCacheAnchorTime;
datetime anchorNow = m_Time.GetData(0);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
if(EnsureBarCachesCapacity(bars) && m_labelCachePrebuilt)
{
fix(diag): the cache-invalidation stall message could never name the cause SP500 and XAUUSD LSTM wedged at era 1 from 19:43 to 21:00+ (77 min) while their three siblings passed era 200 - the 12-minute barrier exclusion correctly kept the charts alive, so the ensembles ran three-handed. Both printed: cache invalidated at era start (era sized 16236 bars, cache holds 16236) Equal numbers, which reads as "so it wasn't the size". That inference is not available: EnsureBarCachesCapacity assigns BOTH invalidation keys (m_labelCacheBars = bars, m_labelCacheAnchorTime = m_Time.GetData(0)) before it returns true, so a message built afterwards reports the values it just overwrote. The two counts are equal BY CONSTRUCTION and ReportTrainStall's anchor= field is always the live one. The line whose stated job is to name which key tripped was structurally incapable of naming it. Capture bars/anchor BEFORE the call and say which one moved: "SIZE CHANGED 16236 -> 16240" or "size unchanged", and "ANCHOR MOVED 2026.08.18 00:00 -> 04:00" or "anchor unchanged". Not guessing at the cause. An anchor moving every era with the size steady is a new candle each pass or a Time buffer that is not being refreshed; a moving size is the era/prebuild disagreement the branch was written for. The next occurrence will say which, instead of costing another session. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 21:57:12 -04:00
//--- The failure mode: the era and the prebuild disagree about `bars`, or about which bar is
//--- index 0, and re-arm each other forever - caches wiped, relabelled, wiped again, no era ever
//--- runs. Name WHICH of the two keys tripped and by how much, or the next occurrence costs
//--- another session to attribute.
string sizeKey = (bars != barsBefore)
? StringFormat("SIZE CHANGED %d -> %d", barsBefore, bars) : "size unchanged";
string anchorKey = (anchorNow != anchorBefore)
? StringFormat("ANCHOR MOVED %s -> %s", TimeToString(anchorBefore),
TimeToString(anchorNow)) : "anchor unchanged";
ReportTrainStall(StringFormat("cache invalidated at era start - %s, %s (era sized %d bars, cache"
" held %d). An anchor that moves EVERY era with the size steady is"
" a new candle each pass or a Time buffer that is not being"
" refreshed; a size that moves is the era/prebuild disagreement.",
sizeKey, anchorKey, bars, barsBefore));
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
StartLabelCachePrebuild();
return;
}
refactor(ai): extract Layer.mqh and deduplicate AI config - Moves CLayer neuron construction to AI/Impl/Layer.mqh to keep Network.mqh clean - Unifies four previously duplicated architecture initialisation blocks (MLP/CONV/LSTM/HYBRID) into a single shared function - Eliminates risk of behavioural drift where one architecture missed a setter, causing mismatched feature sets or targets
2026-08-01 11:27:28 -04:00
//--- freeze the just-finished era's true class totals for this new era's priors (see
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- m_prevEraTrueBuyCount's declaration comment) before resetting the live counters below - EXCEPT
//--- right after StartLabelCachePrebuild()/AdvanceLabelCachePrebuild() seeded them for era 0: the
//--- live m_trueBuyCount/Sell/Neutral tally is still all-zero at that point (nothing trained yet),
refactor(ai): extract Layer.mqh and deduplicate AI config - Moves CLayer neuron construction to AI/Impl/Layer.mqh to keep Network.mqh clean - Unifies four previously duplicated architecture initialisation blocks (MLP/CONV/LSTM/HYBRID) into a single shared function - Eliminates risk of behavioural drift where one architecture missed a setter, causing mismatched feature sets or targets
2026-08-01 11:27:28 -04:00
//--- so copying it here would silently stomp the real upfront tally back to an empty distribution.
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
if(m_prebuildSeedPending)
m_prebuildSeedPending = false;
else
{
m_prevEraTrueBuyCount = m_trueBuyCount;
m_prevEraTrueSellCount = m_trueSellCount;
m_prevEraTrueNeutralCount = m_trueNeutralCount;
}
//--- Natural class base rates for the live logit-adjusted decision (see AdjustedSignalFromSoftmax):
//--- derived from the same just-finished-era true class totals the oversampling ratio uses, so live
//--- calibrates to exactly the distribution the model was measured against. Both branches above
//--- leave m_prevEraTrue* holding the freshest real tally (prebuild-seeded on era 0, copied here
fix: the imbalance correction never ran during the auto-tune search Neutral collapse on all four topologies by era 5 with a 2:6 barrier (recall Buy 0% / Sell 0% / Neutral 100%), and the panel stuck on "measuring...". One root cause, and it was not the barrier. The labels were fine: Buy 25.4% / Sell 22.0% / Neutral 52.5%, which is exactly gambler's ruin for m=2,k=6 (2/8 = 25% per side), with only 0.1% of Neutral coming from the vertical barrier - so the new m*k horizon scaling is right, arguably generous. What was broken: Train()'s era-start block wrapped UpdateClassPriors() in `if(!m_evalMode)`. The auto-tune GA scores every candidate in eval mode, and AutoTuneIndicators ships ON, so on a default configuration EVERY era of the search ran with unmeasured priors. ApplyLogitAdjustment() requires measured priors; without them it calls ClearLogitAdjustment() and returns. So the entire search trained under PLAIN cross-entropy. With a 52.5% majority class the optimum of plain CE is "always predict Neutral", and that is precisely what all four models found. The panel followed: its counters only advance on bars the model CALLED Buy or Sell, so a collapsed model leaves them at zero and the line reads "measuring..." forever. This was latent, not new. It has been true for every auto-tuned run, but it was invisible while the labels were near-balanced - last night's accidental 1:1 barrier gave 43/40/17, where plain CE has no majority to collapse into. Widening the stop to 2*ATR (correctly - 1*ATR is too tight to survive noise) moved Neutral to the majority and exposed it. The guard's stated fear cannot happen. These priors are measured from the LABEL distribution, and the tuner only perturbs indicator periods (MA/RSI/MACD/Ichimoku/AD). The barrier label depends on ATR, SL_Mode and TP_Mode - none of which the search touches - so every candidate sees byte-identical labels and identical priors. There is nothing to contaminate. What the guard actually protected was the .stats write, and that is gated separately: eval candidates never checkpoint and never persist. Also, because this is the THIRD quiet no-op to cost a run in this codebase (after the fictional oversampling log line and the shadow-blend skip): - ApplyLogitAdjustment() now WARNS when it declines to install, instead of silently clearing. A mechanism that cannot announce it is not running is indistinguishable from one that is. - The panel distinguishes "measuring..." (before era 1, nothing scored yet - an honest warm-up) from "no directional calls yet" (eras trained, zero calls - a finding, not a wait). Both builds compile 0 errors / 0 warnings. No retrain forced by this commit itself, but the collapsed models must be discarded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:46:24 -04:00
//--- otherwise), so updating from them here covers both paths.
//--- 2026-08-01: THE `if(!m_evalMode)` GUARD THAT USED TO WRAP THIS IS GONE, because it silently
//--- disabled the entire imbalance correction for the whole auto-tune search. ApplyLogitAdjustment()
//--- immediately below needs measured priors; without them it clears the offsets and returns. In
//--- eval mode the priors were never measured, so every GA candidate - which is to say every era of
//--- a run with AutoTuneIndicators on, the shipped default - trained under PLAIN cross-entropy.
//--- That was invisible while the labels were near-balanced and became a total Neutral collapse the
//--- moment a 2:6 barrier put the majority class at 52.5%: recall Buy 0% / Sell 0% / Neutral 100%
//--- by era 5 on all four topologies, and the panel stuck on "measuring..." because a model that
//--- never calls a direction never accumulates a directional tally.
//--- The guard's stated fear - a search contaminating the deployed calibration - cannot happen:
//--- these priors are measured from the LABEL distribution, and the tuner only perturbs indicator
//--- periods (MA/RSI/MACD/Ichimoku/AD). The barrier label depends on ATR, SL_Mode and TP_Mode, none
//--- of which the search touches, so every candidate sees byte-identical labels and therefore
//--- identical priors. There is nothing for a candidate to contaminate. What the guard actually
//--- protected against is the .stats WRITE, and that is gated separately (eval candidates never
//--- checkpoint - see m_haveOosCheckpoint - and never persist).
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- Meta target trains WITHOUT the logit adjustment, deliberately: the correction exists for
//--- the direction head's extreme class imbalance (directional bars were a ~6% tail), while the
//--- meta label's base rate is the setup's own win rate (~40%), where plain CE is fine and the
//--- operating-point fit (pass 2.5) carries the calibration. Documented deviation from the
//--- design doc's "prior correction" line - the machinery is 3-class-shaped and generalizing it
//--- buys nothing at this base rate.
if(!IsMetaTarget())
{
UpdateClassPriors(m_prevEraTrueBuyCount, m_prevEraTrueSellCount, m_prevEraTrueNeutralCount);
//--- Re-install the training-time logit offsets from the priors just measured, so this
//--- era's gradient tracks the distribution the era is scored against. Runs in eval mode
//--- too: a GA candidate must train under the same loss as the real run or its score
//--- means nothing - only the PERSISTED calibration is withheld from eval mode.
ApplyLogitAdjustment();
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_countBuySignals = 0;
m_countSellSignals = 0;
m_countNeutralSignals = 0;
m_trueBuyCount = 0;
m_trueSellCount = 0;
m_trueNeutralCount = 0;
m_oosBuyHits = 0;
m_oosBuyTotal = 0;
m_oosSellHits = 0;
m_oosSellTotal = 0;
m_oosNeutralHits = 0;
m_oosNeutralTotal = 0;
m_oosBuyPredicted = 0;
m_oosBuyPredictedHits = 0;
m_oosSellPredicted = 0;
m_oosSellPredictedHits = 0;
m_oosNeutralPredicted = 0;
m_oosNeutralPredictedHits = 0;
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
m_oosBuyPredictedWins = 0;
m_oosSellPredictedWins = 0;
feat(gate): grade OOS calls on the exit policy actually in force, and move vote combining out of the members and into the orchestrator Option (a) from the exit-policy question: the certified number must be the traded number. Plus the modularity correction the user called for on 778b6c0. 1. VOTE COMBINING BELONGS TO THE ORCHESTRATOR, NOT TO A MEMBER. 778b6c0 fixed the last-writer-wins bug on g_LiveAISignedConfidence by having a member average its siblings through g_warriorEnsemble. That trades a scheduling bug for a coupling bug, and it is the wrong shape for this EA: every signal runs in its own instance, minds its own state, and VOTES to the orchestrator, which is the only thing allowed to combine opinions. Replaced with a publish/aggregate pair in Variables\ConfidenceBridge.mqh whose split is enforced by shape rather than by convention: - PublishAIVote(slot, conf) - a member writes ONLY its own slot, reads nobody's; - AggregateAIVotes() - called by CExpertSignalCustom::LiveSignedConfidence. CExpertSignalAIBase::EnsembleLiveSignedConfidence is gone. The orchestrator also republishes the aggregate into g_LiveAISignedConfidence, because the intelligent trailing reads that global directly and must act on the same number the exit route does rather than on a leftover from whichever member ticked last. A solo AI signal owns slot 0, so the non-ensemble path is unchanged. 2. THE GATE NOW REPLAYS THE REAL EXIT RULE. SimulateTradeOutcome() walks the same price series with the same fill/barrier/spread convention as ComputeLabelForBar - deliberately by copy, so a disagreement between the two can only be a policy effect and never a discrepancy between two pieces of our own arithmetic - and terminates at the FIRST of stop / target / vote reversal / horizon. Barriers are tested before the vote on the same bar: intrabar we cannot know which came first, and the barrier is what the broker executes automatically, so checking the vote first would credit the exit policy with escapes a real stop would have taken out of its hands. It runs AFTER pass 3, not inside it. A vote-flip exit for a trade entered at bar r is decided by the model's output at bars r-1, r-2, ... - NEWER bars - and pass 3 walks oldest-to-newest, so at the moment r is graded its own exit does not exist yet. Only once m_oosDecisionSeries is complete over the whole OOS window can the replay run. In ensemble mode that series carries the member's adjusted decision and the live exit reads the ensemble aggregate, which is the coupling the user identified: an LSTM entry really can be closed by the ensemble turning against it. 3. THE STATISTIC HAS TO CHANGE WITH THE POLICY, AND THAT IS THE REAL FINDING. A barrier exit pays a fixed R. A vote exit pays whatever the close happens to be. So the moment vote exits are enabled the payoff is CONTINUOUS, and "win rate vs break-even" stops being a meaningful test - there is no fixed break-even for a variable payoff. SimulateTradeOutcome therefore returns R rather than a bool, and the replay reports expectancy in R with its SE taken from the R distribution (overlap- deflated on the same EffectiveSampleSize doctrine as every other SE here), not from a binomial. This is the same class of error as win-based scoring in 2026-08-09: measuring a variable-payoff process with a fixed-payoff statistic. Naming it now, while vote exits are still off, is much cheaper than discovering it after they go on. 4. WHY THIS IS SAFE TO SHIP TODAY. Min_Vote_Close ships Disabled (101, unreachable on both scales it drives) and reaches the AI signal through the new ExitPolicy() setter as 1.01, which the setter turns into 0.0 = no vote exit. Under that policy every replayed trade resolves at a barrier and the simulation is arithmetically the same trade the deploy gate already certifies - they cannot drift. The report says so explicitly, and prints ONCE per run in that state; when vote exits are on it prints every era, because then the divergence is the thing to watch. Nothing about today's numbers moves. The gate switchover is wired but dormant by construction: it becomes exit-aware the moment the input is enabled, which is exactly what "the certified number is the traded number" has to mean. KNOWN LIMIT, stated rather than hidden: only the AI early-exit route is replayed. The rule-based averaged-vote close (m_threshold_close) depends on every other filter's live vote, which pass 3 does not reproduce, so a position the classic filters would have closed is held to its barrier here. The replay therefore holds LONGER than live and overstates barrier-reached outcomes. Faithful only while the AI is the dominant vote - which is the configuration this is being built for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:25:57 -04:00
//--- Simulated-exit accumulators, reset with the rest of the per-era OOS tallies.
m_simRSum = 0.0;
m_simRSumSq = 0.0;
m_simTrades = 0;
m_simVoteExits = 0;
m_simBarrierWins = 0;
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
m_oosWinLongTotal = 0;
m_oosWinShortTotal = 0;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_oosBuyFired = 0;
m_oosBuyFiredHits = 0;
m_oosSellFired = 0;
m_oosSellFiredHits = 0;
fix: NMS gates the TRADE, not just the arrow - one arrow is now one trade NmsLiveAccept() appeared in exactly one place: wrapped around DrawObject(). It never touched dPrevSignal, and dPrevSignal is what LongCondition() / ShortCondition() / SignedAIConfidence() read. So a declustered bar lost its arrow and still opened a position. Measured on SP500 H1 2026-08-09: CONV called a direction on 64% of bars, so the ~500 bars visible on screen held ~320 decisions - and ~40 arrows were drawn. Roughly one arrow per eight positions the EA would take. And the survivors are not a random eighth. Rule 2 of the declustering keeps the HIGHER-CONFIDENCE side of a cluster, so the visible set is systematically the best member of each run. A chart showing the best of every eight decisions and hiding the rest reads far better than the model is - the same best-of-N selection error already corrected in the geometry scan, the indicator tuner, the lag profile and the deploy gate, this time on the display layer, where it is most likely to mislead the person deciding whether to trade. Fixed by neutralising dPrevSignal when NMS rejects, rather than adding a "may trade" flag consulted at each read site: that leaves exactly ONE definition of what the model decided this bar, so the arrow, the panel's "Current signal", the confidence feeding sizing/SL/TP/trailing, the refresh tally and the order itself cannot drift apart again. Also reports the consequence instead of hiding it. Every OOS counter on the era line still scores every directional call - a population ~8x larger than what now trades - so the line carries a second figure: | TRADED (declustered) NN% on N calls (edge +Npp) replaying the identical rule over pass 3 (which walks OOS bars oldest to newest, the same order the live sweep sees). Its cursors are separate members from the live ones so a training pass can never disturb the live chart's declustering. Deliberately NOT switched into selectionScore yet. Declustering cuts coverage from ~64% of bars to ~8%, well under MIN_COVERAGE_FRACTION_OF_BASE_RATE, which would make every checkpoint undeployable overnight - the minRR collision and the recall-floor catch-22 twice over. The floor gets re-derived from these measurements first. Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-09 10:22:31 -04:00
//--- Declustered tally + its replay cursors. -1 / Neutral is "nothing seen yet this era", which is
//--- what makes the first directional call of an era always survive rule 1.
feat: per-family x per-side OOS breakdown in the META era report 350-era S2 verdict on SP500 H1: the meta head carries REAL ranking skill (+1.0-1.3pp mean over base, 101/350 eras clear their own 2-sigma bar, traded subset wins 66.1% at <30% coverage vs 64.5% base) but 0/350 eras produced a positive cov x (p - BE): the candidate stream sits 3pp under the derived geometry's 67.5% break-even and ~2.6pp of recovered skill cannot bridge it. Skill plateaued by mid-run (1.28pp -> 1.05pp), so more eras only buy multiplicity, and the deploy gate correctly shipped nothing. The aggregate can hide a deployable subset (one family/side clearing BE blended with junk), so the META era line now decomposes the SAME traded population into MA/RSI/MACD/Ichimoku x LONG/SHORT cells, each as traded/candidates base->traded win rate. 32 cells is a best-of-N search by construction - any candidate cell faces the family-wise rule before belief. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 09:03:40 -04:00
//--- meta per-family/per-side OOS decomposition - see the member declaration
ArrayInitialize(m_metaFamCand, 0);
ArrayInitialize(m_metaFamWins, 0);
ArrayInitialize(m_metaFamFired, 0);
ArrayInitialize(m_metaFamFiredWins, 0);
ArrayInitialize(m_metaSideCand, 0);
ArrayInitialize(m_metaSideWins, 0);
ArrayInitialize(m_metaSideFired, 0);
ArrayInitialize(m_metaSideFiredWins, 0);
fix: NMS gates the TRADE, not just the arrow - one arrow is now one trade NmsLiveAccept() appeared in exactly one place: wrapped around DrawObject(). It never touched dPrevSignal, and dPrevSignal is what LongCondition() / ShortCondition() / SignedAIConfidence() read. So a declustered bar lost its arrow and still opened a position. Measured on SP500 H1 2026-08-09: CONV called a direction on 64% of bars, so the ~500 bars visible on screen held ~320 decisions - and ~40 arrows were drawn. Roughly one arrow per eight positions the EA would take. And the survivors are not a random eighth. Rule 2 of the declustering keeps the HIGHER-CONFIDENCE side of a cluster, so the visible set is systematically the best member of each run. A chart showing the best of every eight decisions and hiding the rest reads far better than the model is - the same best-of-N selection error already corrected in the geometry scan, the indicator tuner, the lag profile and the deploy gate, this time on the display layer, where it is most likely to mislead the person deciding whether to trade. Fixed by neutralising dPrevSignal when NMS rejects, rather than adding a "may trade" flag consulted at each read site: that leaves exactly ONE definition of what the model decided this bar, so the arrow, the panel's "Current signal", the confidence feeding sizing/SL/TP/trailing, the refresh tally and the order itself cannot drift apart again. Also reports the consequence instead of hiding it. Every OOS counter on the era line still scores every directional call - a population ~8x larger than what now trades - so the line carries a second figure: | TRADED (declustered) NN% on N calls (edge +Npp) replaying the identical rule over pass 3 (which walks OOS bars oldest to newest, the same order the live sweep sees). Its cursors are separate members from the live ones so a training pass can never disturb the live chart's declustering. Deliberately NOT switched into selectionScore yet. Declustering cuts coverage from ~64% of bars to ~8%, well under MIN_COVERAGE_FRACTION_OF_BASE_RATE, which would make every checkpoint undeployable overnight - the minRR collision and the recall-floor catch-22 twice over. The floor gets re-derived from these measurements first. Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-09 10:22:31 -04:00
m_oosNmsFired = 0;
m_oosNmsHits = 0;
m_oosNmsLastBuyIdx = -1;
m_oosNmsLastSellIdx = -1;
m_oosNmsKeptIdx = -1;
m_oosNmsKeptConf = 0.0;
m_oosNmsKeptDir = Neutral;
feat(ai): measure precision per confidence tier; fix stale metric labels Two things the 2026-07-30 run exposed. 1. Every user-facing message still called the selection metric "balanced accuracy". It has ranked on directional precision since a142749, so "CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5% PRECISION as if it were macro-recall, while the same era logged an actual balanced accuracy of 49%. Two different numbers under one name, in the line that announces a deploy. Relabelled at every site, including the stage-3 refusal, which still described the per-class recall floor that stopped being the gate. 2. Precision is now bucketed by confidence tier and logged per era, both per-tier and cumulatively from each tier upward: | tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ... The per-tier number says whether confidence is calibrated to correctness at all; if it does not rise T0->T3, raising the floor buys nothing and that is the finding. The ">=" number is what a floor would actually deliver, with its fire count, so the coverage cost is visible in the same line. Tier weights are 25/50/75/100, so for an AI-only config Min_Vote_Open maps straight across: 50 = ">=T1", 75 = ">=T2", 100 = ">=T3". Bucketing happens at the existing live-fired accounting site, so it measures exactly the population that trades - not the raw argmax. Both builds compile 0 errors, 0 warnings. No retrain needed for either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
ArrayInitialize(m_oosTierFired, 0);
ArrayInitialize(m_oosTierHits, 0);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_oosConfidenceSum = 0;
// Nearest-to-present slice of this era's bars is held out as OOS and never backprop'd on;
// the rest (older bars) is the IS/training slice.
totalIter = (int)MathMax(bars - MathMax(m_historyBars, 0), 0);
oosCutoff = (int)(MathMax(0, MathMin(100, m_oosSplitPct)) / 100.0 * totalIter);
i = (int)(bars - MathMax(m_historyBars, 0) - 1);
//--- Fresh era: reset pass 2's shuffled-backprop queue (see m_isTrainQueue's declaration
//--- comment). Preallocated to a parity-shaped ESTIMATE, not a hard worst case: at full parity
//--- all 3 classes replicate to ~the majority count, so the queue lands near 3x totalIter -
//--- 4x covers that plus label drift. The queueing block below grows the arrays on demand if an
//--- era ever exceeds the estimate (it used to silently DROP overflow instead - harmless at the
//--- old totalIter*cap sizing, which could never fill, but real data loss now that the measured
//--- ratio, not a small fixed cap, decides the replica count).
ArrayResize(m_isTrainQueue, totalIter * 4);
ArrayResize(m_isTrainQueueWeightScale, totalIter * 4);
ArrayResize(m_isTrainQueuePrimary, totalIter * 4);
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
ArrayResize(m_isTrainQueueCand, totalIter * 4);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_isTrainQueueCount = 0;
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
//--- Heartbeat baseline for this era - see the member declarations for why this exists.
m_eraStartTick = GetTickCount();
m_passFeatUs = 0;
m_passNetUs = 0;
diag: an era that discards itself now says so instead of scanning forever add_loop is exactly "at least one bar produced a usable feature window". When it stays false, pass 2, pass 3, the era counter, the checkpoint and every log line in the era-end block are ALL skipped: Train() returns having done nothing, m_eraResumePending is still false, and the next call restarts the SAME era from bar 0. That is an infinite 0->100% "scan" loop that prints absolutely nothing - the only remaining silent restart path in Train(), and it matches the reported symptom exactly. Pass 1 now counts usable vs unusable windows and reports at the pass boundary, which demonstrably executes: - total failure routes through ReportTrainStall (already capped at one line a minute, and carries the run-state flags) naming the counts, the required window width and the bar count - success prints how long the scan took and how many samples it handed to pass 2, but only once the era has passed 10s - a fast era stays as quiet as before, a slow one distinguishes "advancing" from "sweeping the same bars forever" A PARTIAL failure is normal and deliberately does not shout: pass 1 walks oldest-to-newest and the deepest bars predate the indicators' warm-up, so those windows fail and are cached as misses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:29:33 -04:00
m_passWindowOk = 0;
m_passWindowFail = 0;
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
m_passHeartbeatPrints = 0;
diag: the heartbeat could be outrun by the condition it watched for It fired only on 4096-item boundaries once an era had already run 60s. Those boundaries are all crossed in the first few chunks of pass 1, so an era that became slow AFTER them printed nothing at all - which is precisely what happened: 20 minutes, four pegged cores, zero heartbeats. I read that silence as "the era loop is never reached" and went looking for a wedge above it. The silence may simply have meant "past the last boundary". A diagnostic whose trigger can be outrun by the condition it watches for is worse than no diagnostic, because it produces confident wrong conclusions. Now time-gated: checked every 256 items (the mask only keeps GetTickCount off the hot path), prints when the era has run >60s and >30s since the last line, up to 12 per era. Progress/phase for the panel is still published on every call, before any gate. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:44:03 -04:00
m_lastHeartbeatTick = 0;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_isTrainCursor = 0;
m_isPass2Active = false;
m_isPass2Done = false;
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
m_isCalibActive = false;
m_isCalibDone = false;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_isPass3Active = false;
feat: excursion-size head (Stage 1, measurement only) Direction is closed - normalised asymmetry fails on three instruments with a working positive control, and the classifier's own best-of-999 era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is a different question and RANGE clears at ~4x its null. Checked the denomination before building on that, since the source memo warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is predictable" is a claim about travel RELATIVE to current ATR, not a restatement of "ATR is autocorrelated". It is exactly the part a fixed multiple (stop 3.31*ATR, target 1.64*ATR) discards. A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches ladder rung k) upward and downward. Survival parameterisation rather than regressing the multiple, because it needs nothing new from CNet: sigmoid outputs and the per-neuron delta the `total != 3` branch already applies (a quantile head would need a linear activation and a pinball gradient in Network.mqh, Network.cl and the DirectML path, on a class four topologies share). Targets are free - m_ladderUpAt already records first-touch age per rung with 0 meaning never reached. Separate net, not extra outputs on the classifier: more outputs would change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push the count off 3 - the exact condition backProp uses to select the joint softmax gradient the 3-class head depends on. The classifier is bit-for-bit unaffected and this is removable without trace. STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the constant per-rung base rate - the baseline a fixed ATR multiple already assumes - with both predictors fitted IS and evaluated OOS, so neither gets a look at the test set. Positive skill justifies Stage 2 (drive SL/TP and sizing off ExcursionQuantile, which is defined and deliberately uncalled). Zero or negative means ATR already carries everything and Stage 2 must not be built. Trains only on primary occurrences: the replay queue oversamples for CLASS balance, and a direction-balanced sample is a biased SIZE sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
//--- Excursion head: per-era Brier accumulators only. The base rates it is compared against are a
//--- property of the data, not of the era, so they keep accumulating (see ExcursionResetEraScores).
ExcursionResetEraScores();
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- Fresh per-era predicted-signal cache for the end-of-era NMS sweep (see PruneDirectionalClusters).
//--- -2 = "not scored this era" so stale bars from a longer prior era can't draw phantom arrows.
if(m_signalClusterWindow > 0)
{
ArrayResize(m_arrowSignalCache, bars);
ArrayInitialize(m_arrowSignalCache, -2.0);
}
}
else
{
//--- resuming a chunk that yielded mid-bar-loop last call - pick up exactly where it left off
bars = m_resumeBars;
totalIter = m_resumeTotalIter;
oosCutoff = m_resumeOosCutoff;
add_loop = m_resumeAddLoop;
i = m_resumeBarIndex;
m_eraResumePending = false;
}
// Restore this model's own learning-rate trajectory into the shared global right before this
// chunk's backProp() calls touch it - see m_modelEta's declaration comment.
eta = m_modelEta;
uint chunkStartTick = GetTickCount();
// Iterate over the bars - skipped entirely when resuming straight into pass 2, OR when resuming
// into a still-unfinished pass 3 (see m_isPass2Done's declaration comment for why checking
// m_isPass2Active alone isn't enough to detect the latter case): pass 1 already fully completed
// in an earlier call either way.
if(!m_isPass2Active && !m_isPass2Done)
{
for(; i >= 0 && !stop; i--)
{
//--- Build THIS bar's own feature window and feed it forward BEFORE checking/training against
//--- its label - see r's declaration comment below for why the window must end AT bar i, and
//--- why this must run before the label-check block rather than after: the label check needs
//--- this bar's own freshly-computed prediction, not the previous iteration's (see windowOk).
TempData.Clear();
//--- Window ends AT (includes) bar i itself, extending m_historyBars bars into the past - i.e.
//--- "everything known as of this bar's close." Predicting label(i) - "was THIS bar the
//--- reversal" - from a window that stops short of bar i itself would blind the model to the
//--- most recent price action, which is exactly the information a reversal call most depends
fix: live inference queried the 1-tick forming bar - a window training never built RefreshLatestSignal ran at the first tick after a bar opens and built its window at r=0: series index 0 at that instant is a candle with one tick of data - (close-open)/atr ~ 0, high ~ low, degenerate volume, indicators on a 1-tick bar. Training never produces such a window (every labeled bar is fully closed, entry at that bar's CLOSE), so the deployed model's final timestep - the one the LSTM/HYBRID output is keyed to - was out-of-distribution on every live decision, and pass 3's deploy-gate OOS scores measured a different query than live executed. The parity index is r=1: the newest CLOSED bar, whose close IS the current price - the exact instant the label's hypothetical entry happens. Single backtests shared the old skew (same r=0), which is why the tester agreed with live while both disagreed with training. Bookkeeping split that the index change forces: m_lastBarTime/dtStudied stay anchored to the FORMING bar's open (they gate against SERIES_LASTBAR_DATE; anchoring at bar 1 would re-fire the refresh every tick), while bt - the arrow, its High/Low placement, and NMS declustering - anchors to the decision bar, now matching the rescan path's convention. Also: a failed refresh no longer trades the previous bar's signal for the whole bar. RefreshLatestSignal returns success, zeroes dPrevSignal on failure (no opinion beats a stale one), and RefreshConvergedSignal advances dtStudied only on success so the next tick retries - the tester path (m_lastBarTime) already worked this way; this is the live path catching up. FORCES RE-VALIDATION of deployed models: the effective live query distribution changes. Bundled with the backprop transpose fix's retrain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 18:10:23 -04:00
//--- on. Must match RefreshLatestSignal()'s window exactly (r=i there too - live that is
//--- i=1, the newest CLOSED bar, since at the first tick after a bar opens index 0 is a
//--- 1-tick forming candle no training window ever contained; see the 2026-08-11 parity
//--- comment there), since that's what actually queries the deployed model live - training
//--- on a different window than what gets queried at inference time would teach the wrong
//--- task entirely.
fix: the sequence models were reading the window backwards BuildFeatureWindow() replaces eight hand-rolled copies of the same loop and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first, because MQL5 timeseries indices run backwards and `r + b` with b ascending walks into the past. Harmless for PAI and CONV - a dense layer learns a weight per position either way, a conv learns time-mirrored kernels. Not harmless for the recurrent stacks: - LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t. - It writes output[] only when t == steps-1: the visible output IS the last hidden state. - c_t = f*c_{t-1} + i*g decays toward the start of the sequence. lstm_seq_flowcheck.cpp measured block 0's influence on the output at 1.2e-2 of block T-1's, at the shipped forget bias of 1.0. So the bar being PREDICTED sat at the far end of the decay and the output was handed to the OLDEST bar in the window - the exact inverse of what the window is for. ~80x backwards on LSTM and HYBRID, on all three tiers (OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never surfaced as a backend discrepancy. This does not create edge - the MI diagnostics read at the noise floor (p=0.4975) with a working positive control. It makes the one hypothesis those diagnostics explicitly do NOT cover testable: they are marginal and per-bar, and state they "cannot rule out one that only exists in combination or across time". The sequence model is the instrument for across-time structure and it has been crippled, so that hypothesis has never been honestly tested. Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and its features, so a stale .nnw would load cleanly and run a model fitted to one ordering against the other, silently. Re-keying every config is the point, not collateral damage. FORCES A FULL RETRAIN. Also: the now-relative bar caches are re-keyed on the two live paths. EnsureBarCachesCapacity() was only ever called from training paths, but once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar to RefreshConvergedSignal() and Train() is never re-entered - so nothing cleared the feature cache again for the life of the process. A chart that trained to convergence kept replaying the rows computed for the last training era's bar grid: the live signal froze at its convergence-time value, and OnlineLearnStep() backpropped those stale features against freshly resolved labels. Backtests were never affected (an inference-only process never allocates the arrays, so every read recomputes). Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 18:28:44 -04:00
//--- BuildFeatureWindow() owns the Clear/Reserve/loop AND the oldest-bar-first ordering that
//--- the LSTM stacks depend on - see its definition comment.
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
int r = i;
bool windowOk = false;
double displayNeuron0 = 0, displayNeuron1 = 0, displayNeuron2 = 0;
if(r <= bars)
{
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
ulong hbT = GetMicrosecondCount();
fix: the sequence models were reading the window backwards BuildFeatureWindow() replaces eight hand-rolled copies of the same loop and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first, because MQL5 timeseries indices run backwards and `r + b` with b ascending walks into the past. Harmless for PAI and CONV - a dense layer learns a weight per position either way, a conv learns time-mirrored kernels. Not harmless for the recurrent stacks: - LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t. - It writes output[] only when t == steps-1: the visible output IS the last hidden state. - c_t = f*c_{t-1} + i*g decays toward the start of the sequence. lstm_seq_flowcheck.cpp measured block 0's influence on the output at 1.2e-2 of block T-1's, at the shipped forget bias of 1.0. So the bar being PREDICTED sat at the far end of the decay and the output was handed to the OLDEST bar in the window - the exact inverse of what the window is for. ~80x backwards on LSTM and HYBRID, on all three tiers (OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never surfaced as a backend discrepancy. This does not create edge - the MI diagnostics read at the noise floor (p=0.4975) with a working positive control. It makes the one hypothesis those diagnostics explicitly do NOT cover testable: they are marginal and per-bar, and state they "cannot rule out one that only exists in combination or across time". The sequence model is the instrument for across-time structure and it has been crippled, so that hypothesis has never been honestly tested. Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and its features, so a stale .nnw would load cleanly and run a model fitted to one ordering against the other, silently. Re-keying every config is the point, not collateral damage. FORCES A FULL RETRAIN. Also: the now-relative bar caches are re-keyed on the two live paths. EnsureBarCachesCapacity() was only ever called from training paths, but once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar to RefreshConvergedSignal() and Train() is never re-entered - so nothing cleared the feature cache again for the life of the process. A chart that trained to convergence kept replaying the rows computed for the last training era's bar grid: the live signal froze at its convergence-time value, and OnlineLearnStep() backpropped those stale features against freshly resolved labels. Backtests were never affected (an inference-only process never allocates the arrays, so every read recomputes). Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 18:28:44 -04:00
windowOk = BuildFeatureWindow(r);
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
m_passFeatUs += GetMicrosecondCount() - hbT;
fix: the sequence models were reading the window backwards BuildFeatureWindow() replaces eight hand-rolled copies of the same loop and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first, because MQL5 timeseries indices run backwards and `r + b` with b ascending walks into the past. Harmless for PAI and CONV - a dense layer learns a weight per position either way, a conv learns time-mirrored kernels. Not harmless for the recurrent stacks: - LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t. - It writes output[] only when t == steps-1: the visible output IS the last hidden state. - c_t = f*c_{t-1} + i*g decays toward the start of the sequence. lstm_seq_flowcheck.cpp measured block 0's influence on the output at 1.2e-2 of block T-1's, at the shipped forget bias of 1.0. So the bar being PREDICTED sat at the far end of the decay and the output was handed to the OLDEST bar in the window - the exact inverse of what the window is for. ~80x backwards on LSTM and HYBRID, on all three tiers (OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never surfaced as a backend discrepancy. This does not create edge - the MI diagnostics read at the noise floor (p=0.4975) with a working positive control. It makes the one hypothesis those diagnostics explicitly do NOT cover testable: they are marginal and per-bar, and state they "cannot rule out one that only exists in combination or across time". The sequence model is the instrument for across-time structure and it has been crippled, so that hypothesis has never been honestly tested. Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and its features, so a stale .nnw would load cleanly and run a model fitted to one ordering against the other, silently. Re-keying every config is the point, not collateral damage. FORCES A FULL RETRAIN. Also: the now-relative bar caches are re-keyed on the two live paths. EnsureBarCachesCapacity() was only ever called from training paths, but once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar to RefreshConvergedSignal() and Train() is never re-entered - so nothing cleared the feature cache again for the life of the process. A chart that trained to convergence kept replaying the rows computed for the last training era's bar grid: the live signal froze at its convergence-time value, and OnlineLearnStep() backpropped those stale features against freshly resolved labels. Backtests were never affected (an inference-only process never allocates the arrays, so every read recomputes). Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 18:28:44 -04:00
if(windowOk)
diag: an era that discards itself now says so instead of scanning forever add_loop is exactly "at least one bar produced a usable feature window". When it stays false, pass 2, pass 3, the era counter, the checkpoint and every log line in the era-end block are ALL skipped: Train() returns having done nothing, m_eraResumePending is still false, and the next call restarts the SAME era from bar 0. That is an infinite 0->100% "scan" loop that prints absolutely nothing - the only remaining silent restart path in Train(), and it matches the reported symptom exactly. Pass 1 now counts usable vs unusable windows and reports at the pass boundary, which demonstrably executes: - total failure routes through ReportTrainStall (already capped at one line a minute, and carries the run-state flags) naming the counts, the required window width and the bar count - success prints how long the scan took and how many samples it handed to pass 2, but only once the era has passed 10s - a fast era stays as quiet as before, a slow one distinguishes "advancing" from "sweeping the same bars forever" A PARTIAL failure is normal and deliberately does not shout: pass 1 walks oldest-to-newest and the deepest bars predate the indicators' warm-up, so those windows fail and are cached as misses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:29:33 -04:00
{
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
add_loop = true;
diag: an era that discards itself now says so instead of scanning forever add_loop is exactly "at least one bar produced a usable feature window". When it stays false, pass 2, pass 3, the era counter, the checkpoint and every log line in the era-end block are ALL skipped: Train() returns having done nothing, m_eraResumePending is still false, and the next call restarts the SAME era from bar 0. That is an infinite 0->100% "scan" loop that prints absolutely nothing - the only remaining silent restart path in Train(), and it matches the reported symptom exactly. Pass 1 now counts usable vs unusable windows and reports at the pass boundary, which demonstrably executes: - total failure routes through ReportTrainStall (already capped at one line a minute, and carries the run-state flags) naming the counts, the required window width and the bar count - success prints how long the scan took and how many samples it handed to pass 2, but only once the era has passed 10s - a fast era stays as quiet as before, a slow one distinguishes "advancing" from "sweeping the same bars forever" A PARTIAL failure is normal and deliberately does not shout: pass 1 walks oldest-to-newest and the deepest bars predate the indicators' warm-up, so those windows fail and are cached as misses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:29:33 -04:00
m_passWindowOk++;
}
else
m_passWindowFail++;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
fix: the panel showed "100%" for the whole of pass 1 The simple panel derived its percentage from pass 2's counters: (m_isTrainCursor+1) / max(m_isTrainQueueCount,1). During pass 1 those are 0 and 0, so the expression is (0+1)/max(0,1) = 100%. An era spends its first pass scanning ~38k bars - minutes of work - and the panel reported that phase as finished the entire time. Observed by the user as "started learning at 100% of their era and are stuck there", and it actively misled the diagnosis: the one number on screen said the opposite of what was happening. The UI cannot fix this on its own - it can see pass 2's counters but has no way to know which pass owns them. So each pass now PUBLISHES its own progress and a short phase name through TrainHeartbeat (which every pass already calls per item), and the panel just displays them: "learning (era 45, scan 34%)". Published before the heartbeat's 4096-item journal gate, so the panel updates continuously while the journal stays quiet. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:22:29 -04:00
TrainHeartbeat("pass 1 (scan/queue), bar", bars - MathMax(m_historyBars, 0) - i, totalIter, "scan");
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- Determine label/queue-eligibility BEFORE running any feedForward this bar - see
//--- wouldQueue's use below for why. Mirrors the label-check condition this block used to
//--- gate on (moved earlier, unchanged).
bool haveLabel = false, buy = false, sell = false, wouldQueue = false;
perf: pass 1 forward-passed ~40% of bars that a later pass redid anyway Pass 1 already skipped its feedForward on QUEUED bars, because pass 2 redoes them. The same argument covers two more bands it was still forwarding: OOS window (30% of bars) - pass 3 re-forwards every one of them calibration band (~10% of bars) - pass 2.5 re-forwards every one of them All three passes derive their bounds from the same helpers and apply the identical eligibility test, so the bar sets are equal by construction, not by coincidence. Only the two purge bands and the ineligible edge bars are visited in pass 1 and nowhere else - those keep their forward pass. The scan's copy was never the one that survived. Its arrow-cache write was overwritten by pass 3's (with the thresholded, post-training decision), its status-label paint was transient, and its predicted-class tally measured last era's weights. Those tallies move to pass 2.5 and pass 3, on the raw argmax exactly as pass 1 and pass 2 count it, so the population behind the panel's "Predicted -> Buy/Sell/Neutral" line is unchanged and stays comparable with the "Actual" line beside it, which pass 1 still accumulates over every labelled bar. Verified unaffected by the cut: dPrevSignal and m_lastBarTime are both written last by bars 0/1, which are label-ineligible and therefore still forwarded, so FinalizeTrainRun's `dtStudied = m_lastBarTime` and Lifecycle's newBarPending sentinel read the same values as before. Correctness, not just speed: batch norm is UNFROZEN during pass 1 (passes 2.5 and 3 freeze it deliberately), so every scan-time forward on a held-out bar was advancing the BN running mean/variance from data the model is graded on. Those running statistics are inference-time model state. It is the mild, unsupervised kind of leakage - feature statistics, not labels - but it fed the weights pass 3 then scored, and it is now gone. Cost: ~40% of all bars lose one forward pass per era, ~16% of net time once pass 2's backward pass is weighted in. Per-dispatch, so it lands on every backend. Both variants compile 0 errors / 0 warnings. Build tag scan-nofwd-v5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:50:14 -04:00
//--- "some LATER pass in this same era will feed this exact bar forward anyway", which is a
//--- strictly wider set than wouldQueue - see its use at the feedForward below. Declared out
//--- here because the three membership tests that decide it are scoped to the label block.
bool laterPassForwards = false;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
if(windowOk && i < (int)(bars - MathMax(m_historyBars, 0) - 1) && i > 1 && m_Time.GetData(i) > dtStudied
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
&& (m_outputNeuronsCount == 1 || m_outputNeuronsCount == 3 || IsMetaTarget()))
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
//--- The fractal/swing-confirmation/trend-context label at now-relative index i only depends
//--- on price/ATR history, never on model state, so it's identical every era until a new bar
//--- closes and shifts the index frame (see the cache invalidation check above) - cache it
//--- rather than recomputing from scratch every single era. Usually already populated by
//--- AdvanceLabelCachePrebuild() before era 0 ever starts - this is just a lazy fallback for
//--- any index it didn't cover (e.g. bars/window drifted between prebuild and era 0's start).
if(m_labelCacheHasValue[i])
{
buy = m_labelCacheBuy[i];
sell = m_labelCacheSell[i];
}
else
{
ComputeLabelForBar(i, bars, buy, sell);
m_labelCacheBuy[i] = buy;
m_labelCacheSell[i] = sell;
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
//--- Kept in step with the label caches by hand here, because this fallback does not go
//--- through AdvanceBarrierLabelState. ComputeLabelForBar is a stub that returns no label,
//--- so "no winning direction" is the honest entry - but leaving them unwritten would mean
//--- reading whatever ArrayResize left behind, under a validity flag that says otherwise.
if(i < ArraySize(m_winLongCache))
{
m_winLongCache[i] = false;
m_winShortCache[i] = false;
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_labelCacheHasValue[i] = true;
}
haveLabel = true;
bool isOOS = (i < oosCutoff);
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
// Embargo: a bar's triple-barrier label is decided by the m_barrierHorizonBars bars that
// follow it (see TripleBarrierLabel()). An IS bar within that distance of the OOS boundary
// therefore carries a label that was only knowable using price action from inside the
// held-out OOS window - purge that narrow band from backprop entirely instead of training
// on it as ordinary IS. Lopez de Prado ch. 7 calls this purging, and it is the whole reason
// a naive train/test split leaks on overlapping-horizon financial labels.
// Was m_swingConfirmationBars + LABEL_WINDOW_BARS, which measured the ZigZag repainting
// delay - the correct quantity for the old target and the wrong one for this label.
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
int calibLo = CalibLoIndex(oosCutoff); // = oosCutoff + one purge width
int calibHi = CalibHiIndex(totalIter, oosCutoff); // == calibLo when the band is empty
bool isEmbargoed = (!isOOS && i < calibLo);
//--- The calibration slice and its far-side purge are held out of backprop for the same
//--- reason the OOS window is, and the layout is documented once at CalibLoIndex(). This is
//--- the ONLY place the band is excluded from training - the walk that scores it (pass 2.5)
//--- derives its bounds from the same two helpers, so the two cannot disagree about which
//--- bars are held out.
bool isCalib = (i >= calibLo && i < calibHi);
bool isCalibPurge = (calibHi > calibLo && i >= calibHi && i < calibHi + CalibPurgeBars());
wouldQueue = (!isOOS && !isEmbargoed && !isCalib && !isCalibPurge);
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- Meta target: only bars HOSTING a candidate carry training rows, and passes 2/2.5/3
//--- forward those per-candidate themselves (the descriptor differs per candidate, so a
//--- bar-level scan forward could not be reused anyway). Everything scan-side that reads a
//--- forward pass is direction-display machinery, so the meta path skips it entirely.
wouldQueue = wouldQueue && (!IsMetaTarget() || MetaCandFirst(i) >= 0);
perf: pass 1 forward-passed ~40% of bars that a later pass redid anyway Pass 1 already skipped its feedForward on QUEUED bars, because pass 2 redoes them. The same argument covers two more bands it was still forwarding: OOS window (30% of bars) - pass 3 re-forwards every one of them calibration band (~10% of bars) - pass 2.5 re-forwards every one of them All three passes derive their bounds from the same helpers and apply the identical eligibility test, so the bar sets are equal by construction, not by coincidence. Only the two purge bands and the ineligible edge bars are visited in pass 1 and nowhere else - those keep their forward pass. The scan's copy was never the one that survived. Its arrow-cache write was overwritten by pass 3's (with the thresholded, post-training decision), its status-label paint was transient, and its predicted-class tally measured last era's weights. Those tallies move to pass 2.5 and pass 3, on the raw argmax exactly as pass 1 and pass 2 count it, so the population behind the panel's "Predicted -> Buy/Sell/Neutral" line is unchanged and stays comparable with the "Actual" line beside it, which pass 1 still accumulates over every labelled bar. Verified unaffected by the cut: dPrevSignal and m_lastBarTime are both written last by bars 0/1, which are label-ineligible and therefore still forwarded, so FinalizeTrainRun's `dtStudied = m_lastBarTime` and Lifecycle's newBarPending sentinel read the same values as before. Correctness, not just speed: batch norm is UNFROZEN during pass 1 (passes 2.5 and 3 freeze it deliberately), so every scan-time forward on a held-out bar was advancing the BN running mean/variance from data the model is graded on. Those running statistics are inference-time model state. It is the mild, unsupervised kind of leakage - feature statistics, not labels - but it fed the weights pass 3 then scored, and it is now gone. Cost: ~40% of all bars lose one forward pass per era, ~16% of net time once pass 2's backward pass is weighted in. Per-dispatch, so it lands on every backend. Both variants compile 0 errors / 0 warnings. Build tag scan-nofwd-v5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:50:14 -04:00
//--- Pass 2 re-forwards every queued bar, pass 2.5 re-forwards the whole calibration band,
//--- and pass 3 re-forwards the whole OOS window - each over EXACTLY this bar set (all
//--- three derive their bounds from the same helpers and apply the identical eligibility
//--- test this block gates on). Only the two purge bands and the ineligible edge bars are
//--- visited here and nowhere else, so those are the only ones that still need a scan-time
//--- forward pass. At the shipped 30% OOS / 15% calibration split that is ~40% of all bars
//--- whose forward pass was being computed twice per era and thrown away the first time.
laterPassForwards = (wouldQueue || isOOS || isCalib);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
//--- Only run this bar's feedForward (and the display/count/chart-draw work that depends on
perf: pass 1 forward-passed ~40% of bars that a later pass redid anyway Pass 1 already skipped its feedForward on QUEUED bars, because pass 2 redoes them. The same argument covers two more bands it was still forwarding: OOS window (30% of bars) - pass 3 re-forwards every one of them calibration band (~10% of bars) - pass 2.5 re-forwards every one of them All three passes derive their bounds from the same helpers and apply the identical eligibility test, so the bar sets are equal by construction, not by coincidence. Only the two purge bands and the ineligible edge bars are visited in pass 1 and nowhere else - those keep their forward pass. The scan's copy was never the one that survived. Its arrow-cache write was overwritten by pass 3's (with the thresholded, post-training decision), its status-label paint was transient, and its predicted-class tally measured last era's weights. Those tallies move to pass 2.5 and pass 3, on the raw argmax exactly as pass 1 and pass 2 count it, so the population behind the panel's "Predicted -> Buy/Sell/Neutral" line is unchanged and stays comparable with the "Actual" line beside it, which pass 1 still accumulates over every labelled bar. Verified unaffected by the cut: dPrevSignal and m_lastBarTime are both written last by bars 0/1, which are label-ineligible and therefore still forwarded, so FinalizeTrainRun's `dtStudied = m_lastBarTime` and Lifecycle's newBarPending sentinel read the same values as before. Correctness, not just speed: batch norm is UNFROZEN during pass 1 (passes 2.5 and 3 freeze it deliberately), so every scan-time forward on a held-out bar was advancing the BN running mean/variance from data the model is graded on. Those running statistics are inference-time model state. It is the mild, unsupervised kind of leakage - feature statistics, not labels - but it fed the weights pass 3 then scored, and it is now gone. Cost: ~40% of all bars lose one forward pass per era, ~16% of net time once pass 2's backward pass is weighted in. Per-dispatch, so it lands on every backend. Both variants compile 0 errors / 0 warnings. Build tag scan-nofwd-v5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:50:14 -04:00
//--- it) when NO later pass is about to redo it anyway. A bar that a later pass revisits gets
//--- a completely fresh feedForward within this same era, and that later result is strictly
//--- better than this one: it is computed against weights this era has actually trained,
//--- whereas the scan runs before pass 2 has taken a single step. So the scan's copy was never
//--- the one that survived - it was overwritten (arrow cache, status label) or measured a
//--- one-era-stale model (the predicted-class tally), and it cost a full forward pass per bar
//--- to produce. The book's SGD (references\neuronetworksbook.pdf, section 1.4) is one
//--- forward+backward pass per training sample, not two, and the same logic extends to the
//--- held-out bars: one forward pass per SCORED bar, taken by the pass that scores it.
//--- The counters and the arrow-cache write this used to perform for those bars now happen in
//--- pass 2 (queued), pass 2.5 (calibration band) and pass 3 (OOS) respectively, so the
//--- populations behind them are unchanged - only the weights they are measured against are,
//--- and those move from pre-training to post-training, which is the honest reading.
fix: improve error messages and suppress false sharing-violation logs - BufferDouble: replace hardcoded "DirectML/CPU-DLL" with dynamic backend name and add buffer index/element count to all error prints for easier debugging. - NetPersistence: distinguish missing file from transient lock by probing FileIsExist before logging, eliminating false "sharing violation" warnings when no saved model exists on first run.
2026-08-02 01:09:18 -04:00
//--- Display/IS-scoring only on this path, but the same rule applies: getResults() after a
//--- failed pass returns the previous bar's activations, which would be shown on the panel and
//--- counted as this bar's prediction.
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
ulong hbFwd = GetMicrosecondCount();
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- !IsMetaTarget(): the scan-time forward exists only for the direction display/tally on
//--- bars no later pass revisits; a meta forward without a candidate descriptor would be
//--- width-mismatched against the meta input layer as well as meaningless.
bool scanForwardOk = (windowOk && !laterPassForwards && !IsMetaTarget() && Net.feedForward(TempData));
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
m_passNetUs += GetMicrosecondCount() - hbFwd;
if(scanForwardOk)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
Net.getResults(TempData);
if(m_outputNeuronsCount == 1)
dPrevSignal = TempData[0];
else
if(m_outputNeuronsCount == 3)
dPrevSignal = ApplyClassificationSoftmax();
//--- Snapshot the just-computed neuron output(s) for the status label display below, before
//--- the label-check block clears/refills TempData with the target label (Step A always
//--- runs after this point now) - reading TempData directly for display after that would
//--- show the TRUE LABEL of the bar just trained on, not the network's own prediction.
if(TempData.Total() > 0)
displayNeuron0 = TempData[0];
if(TempData.Total() > 1)
displayNeuron1 = TempData[1];
if(TempData.Total() > 2)
displayNeuron2 = TempData[2];
switch(DoubleToSignal(dPrevSignal))
{
case Buy:
m_countBuySignals++;
break;
case Sell:
m_countSellSignals++;
break;
default:
m_countNeutralSignals++;
break;
}
m_lastBarTime = m_Time.GetData(i);
if(i > 0)
{
// NMS on: record only - the era-end sweep is the SOLE renderer, so no raw (un-
// declustered) arrow is ever drawn mid-era. NMS off: draw inline as before.
if(m_signalClusterWindow > 0)
{
if(i < ArraySize(m_arrowSignalCache))
m_arrowSignalCache[i] = dPrevSignal;
}
else
if(DoubleToSignal(dPrevSignal) == Neutral)
DeleteObject(m_lastBarTime);
else
feat(chart): signal marks become price LEVELS at the trigger, not arrows beside the candle User request: 'move from arrows on lows and highs to small horizontal lines at the actual prices the entry/exit would trigger, just a bit larger than the candles. dark green for buy, dark red for sell.' Every mark is now an OBJ_TREND segment with both anchors at one price and both rays off, spanning 1.3 bar widths, drawn at the bar's CLOSE - the price a market order actually fires at, and the exact entry TripleBarrierLabel assumes. It used to sit on the candle's LOW for a Buy and its HIGH for a Sell: prices the trade never touches, picked so an arrow glyph would clear the candle. The tooltip now carries that price too. COLOUR NOW MEANS DIRECTION AND ONLY DIRECTION on every layer (dark green / dark red). Layer moves to width+style - the traded vote is solid and thick and drawn in front, a single model's raw opinion is thin, dotted and behind the candles - which keeps the distinction the old palette existed to draw (a model's opinion must never read as a trade) while freeing colour to say one thing consistently. Consequences handled, all of them the same 'a typed scan went blind' failure: - SaveChartSignals filtered OBJPROP_TYPE == OBJ_ARROW and read OBJPROP_ARROWCODE. It now filters OBJ_TREND and recovers direction from the colour. The sidecar keeps the old 217/218 numbers as its buy/sell token deliberately, so existing .arrows files still load. - AdvanceChartSignalRestore now rebuilds through the SAME creation point the live path uses, so a restored mark and a fresh one are identical objects. - The rescan-scoped delete enumerated ObjectsTotal(OBJ_ARROW) - retyped, or it silently deletes nothing. - ApplySignalsVisibility enumerated OBJ_ARROW with NO prefix filter. Under the new type that would have hidden and shown THE USER'S OWN trend lines on every Hide/Show click; it is now prefix-scoped. The old type was uncommon enough on a real chart to mask the missing check - trend lines are the most hand-drawn object there is. - DrawObject's high/low parameters are gone (6 call sites pass m_Close instead), so no caller can hand it a price it no longer draws at. - Fixed a pre-existing stale comment that still described the purge sweep as OBJ_ARROW-only three lines above the note explaining it had been widened to every type. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 14:11:49 -04:00
DrawObject(m_lastBarTime, dPrevSignal, m_Close.GetData(i));
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
UpdateTrainingStatusLabel(
StringFormat("Bar %d of %d -> %.2f%% (scan)", bars - i + 1, bars, (double)(bars - i + 1.0) / bars * 100),
displayNeuron0, displayNeuron1, displayNeuron2, dPrevSignal);
}
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
else
perf: pass 1 forward-passed ~40% of bars that a later pass redid anyway Pass 1 already skipped its feedForward on QUEUED bars, because pass 2 redoes them. The same argument covers two more bands it was still forwarding: OOS window (30% of bars) - pass 3 re-forwards every one of them calibration band (~10% of bars) - pass 2.5 re-forwards every one of them All three passes derive their bounds from the same helpers and apply the identical eligibility test, so the bar sets are equal by construction, not by coincidence. Only the two purge bands and the ineligible edge bars are visited in pass 1 and nowhere else - those keep their forward pass. The scan's copy was never the one that survived. Its arrow-cache write was overwritten by pass 3's (with the thresholded, post-training decision), its status-label paint was transient, and its predicted-class tally measured last era's weights. Those tallies move to pass 2.5 and pass 3, on the raw argmax exactly as pass 1 and pass 2 count it, so the population behind the panel's "Predicted -> Buy/Sell/Neutral" line is unchanged and stays comparable with the "Actual" line beside it, which pass 1 still accumulates over every labelled bar. Verified unaffected by the cut: dPrevSignal and m_lastBarTime are both written last by bars 0/1, which are label-ineligible and therefore still forwarded, so FinalizeTrainRun's `dtStudied = m_lastBarTime` and Lifecycle's newBarPending sentinel read the same values as before. Correctness, not just speed: batch norm is UNFROZEN during pass 1 (passes 2.5 and 3 freeze it deliberately), so every scan-time forward on a held-out bar was advancing the BN running mean/variance from data the model is graded on. Those running statistics are inference-time model state. It is the mild, unsupervised kind of leakage - feature statistics, not labels - but it fed the weights pass 3 then scored, and it is now gone. Cost: ~40% of all bars lose one forward pass per era, ~16% of net time once pass 2's backward pass is weighted in. Per-dispatch, so it lands on every backend. Both variants compile 0 errors / 0 warnings. Build tag scan-nofwd-v5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:50:14 -04:00
//--- Bars a later pass will re-forward skip the feedForward above, and they are now very
//--- nearly ALL of pass 1 - the queued IS bars (~58%, processed FIRST because the loop
//--- walks oldest-to-newest), plus the calibration band and the OOS slice. Painting only on
//--- the forward path meant the panel sat on the idle writer's "Getting ready..." for the
//--- whole IS sweep, which on a slow era reads exactly like a hang (2026-08-10: four
//--- charts, 20+ minutes, no sign of life anywhere) - and after this change that would be
//--- the ENTIRE scan. The label is throttled internally, so painting every bar costs
//--- nothing.
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
UpdateTrainingStatusLabel(
StringFormat("Bar %d of %d -> %.2f%% (scan)", bars - i + 1, bars, (double)(bars - i + 1.0) / bars * 100),
displayNeuron0, displayNeuron1, displayNeuron2, dPrevSignal);
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- META TARGET: one training row per candidate journaled at this bar (bars without a
//--- candidate carry no rows - wouldQueue already required one). The win/loss label maps onto
//--- the Buy/Sell class-tally slots - THE MAPPING THE WHOLE META PATH RUNS ON: win->Buy,
//--- loss->Sell, Neutral unused. Under it every downstream consumer keeps its meaning with no
//--- era-end changes at all: "buy recall" reads as sensitivity, "sell recall" as specificity
//--- (so bothSidesLive rejects an always-call/never-call collapse), m_oosWinLongTotal/eraBars
//--- becomes the base win rate - which IS the zero-skill precision of calling every candidate
//--- - and coverage becomes the fraction of candidates traded. See the pass 3 meta branch.
if(haveLabel && IsMetaTarget())
{
for(int cd = MetaCandFirst(i); cd >= 0; cd = MetaCandNext(cd))
{
if(MetaCandidateWon(cd, i))
m_trueBuyCount++;
else
m_trueSellCount++;
if(!wouldQueue)
continue;
if(m_isTrainQueueCount + 1 > ArraySize(m_isTrainQueue))
{
int newQueueSize = m_isTrainQueueCount + 1;
ArrayResize(m_isTrainQueue, newQueueSize, 16384);
ArrayResize(m_isTrainQueueWeightScale, newQueueSize, 16384);
ArrayResize(m_isTrainQueuePrimary, newQueueSize, 16384);
ArrayResize(m_isTrainQueueCand, newQueueSize, 16384);
}
m_isTrainQueue[m_isTrainQueueCount] = i;
//--- no oversampling and no per-sample reweighting for the meta label (~40% base rate)
m_isTrainQueueWeightScale[m_isTrainQueueCount] = 1.0;
m_isTrainQueuePrimary[m_isTrainQueueCount] = true;
m_isTrainQueueCand[m_isTrainQueueCount] = cd;
m_isTrainQueueCount++;
}
}
else
if(haveLabel)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
// True label as an ENUM_SIGNAL, derived directly from the buy/sell bools - not read
// back from TempData, which no longer holds a target at this point at all (see above).
ENUM_SIGNAL trueSignal = buy ? Buy : (sell ? Sell : Neutral);
// Track the true class distribution this era (used below to weight IS oversampling,
// and surfaced in the status label text alongside the predicted-class counts)
switch(trueSignal)
{
case Buy:
m_trueBuyCount++;
break;
case Sell:
m_trueSellCount++;
break;
default:
m_trueNeutralCount++;
break;
}
// OOS scoring used to happen right here, against whatever weights this bar's earlier
// feedForward (this pass) happened to be using - which for era 0 is the network's
// still-untrained cold-start state (100% Neutral - see the output-layer bias seed's
// declaration comment), and for every later era is last era's END-of-training state,
// never THIS era's. That silently gave every era's OOS score a full one-era lag behind
// its own training, and made era 0's OOS score meaningless by construction. OOS scoring
// now happens in its own pass (see m_isPass3Active's declaration comment), AFTER pass 2
// has actually trained on this era's IS data, against a fresh feedForward on each OOS
// bar rather than this scan's now-stale one.
if(wouldQueue)
{
// Queue this bar for pass 2's shuffled backProp instead of training on it here,
// immediately, in strict chronological order - see m_isTrainQueue's declaration
refactor(ai): nine class-imbalance inputs down to two The imbalance section offered nine controls for one job. Audited against the code, five of them did not do what their names said at the shipped defaults: AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns whenever the adjusted loss is on, which is default. OversampleParity DEAD in training - Training.mqh gated the replay loop on !useLogitAdjustedLoss (correctly, citing Buda et al. 2018). Live only in the online-learning path. EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma damper - "replay minority bars through pass-2 oversampling" was a focal-loss switch. ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25. UseStaticPrior An exact duplicate of FreezePriorCalibration - the two were OR'd together in the single place either is read. So they were not five mechanisms fighting; they were one mechanism plus eight knobs that mostly described machinery that no longer ran. That is worse than a real conflict, because the log agreed with the names: the label-cache line printed "reps up to 28x (90% parity) (seeding era 0's class-balance oversampling)" on every run, describing an oversampling pass that had been switched off. It is fixed here too - it cost this session a wrong diagnosis. The one genuine redundancy was focal loss, running at gamma*0.125 alongside the adjusted loss: two corrections on the same axis, the exact stacking failure this file already cited Buda et al. for in two other places, damped by a replay flag whose replay path was itself dead. Removed rather than re-tuned. The plateau ladder is unaffected - its escape is the learning-rate warm restart; the gamma anneal beside it only ever stepped toward zero. WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze: LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted- Loss boolean, since a strength dial where 0 already means off does not need an on/off switch beside it. FreezePriorCalibration unchanged. It is the only one of the six corrections with a consistency guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection already ranks on - so the loss and the deploy decision optimize one thing. The online continual-learning path keeps its own alpha-balanced focal weight, now as constants pinned to the removed inputs' shipped defaults, so its behaviour is unchanged. It legitimately needs its own correction: ApplyLogitAdjustment() only runs inside a training run, so a deployed model that was reloaded carries no logit offsets and would otherwise stream 31:1 data into itself uncorrected. The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a double fed to a %d conversion and had always emitted a literal 0; the |MR: segment is written as the constant its shipped defaults produced. Dropping either would have re-keyed every model and forced a from-scratch retrain of the one topology currently converged and trading. Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable "neutralized by prior correction" diagnostic. Both builds compile 0 errors, 0 warnings. No retrain forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
// comment for the full rationale. The predicted-signal counts, the chart-marker draw,
// and the dForecast/dUndefine IS-accuracy update are all computed in pass 2 instead,
// against that bar's own freshly-recomputed confidence - see the matching block right
// after pass 2's Net.feedForward() call.
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//
refactor(ai): nine class-imbalance inputs down to two The imbalance section offered nine controls for one job. Audited against the code, five of them did not do what their names said at the shipped defaults: AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns whenever the adjusted loss is on, which is default. OversampleParity DEAD in training - Training.mqh gated the replay loop on !useLogitAdjustedLoss (correctly, citing Buda et al. 2018). Live only in the online-learning path. EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma damper - "replay minority bars through pass-2 oversampling" was a focal-loss switch. ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25. UseStaticPrior An exact duplicate of FreezePriorCalibration - the two were OR'd together in the single place either is read. So they were not five mechanisms fighting; they were one mechanism plus eight knobs that mostly described machinery that no longer ran. That is worse than a real conflict, because the log agreed with the names: the label-cache line printed "reps up to 28x (90% parity) (seeding era 0's class-balance oversampling)" on every run, describing an oversampling pass that had been switched off. It is fixed here too - it cost this session a wrong diagnosis. The one genuine redundancy was focal loss, running at gamma*0.125 alongside the adjusted loss: two corrections on the same axis, the exact stacking failure this file already cited Buda et al. for in two other places, damped by a replay flag whose replay path was itself dead. Removed rather than re-tuned. The plateau ladder is unaffected - its escape is the learning-rate warm restart; the gamma anneal beside it only ever stepped toward zero. WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze: LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted- Loss boolean, since a strength dial where 0 already means off does not need an on/off switch beside it. FreezePriorCalibration unchanged. It is the only one of the six corrections with a consistency guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection already ranks on - so the loss and the deploy decision optimize one thing. The online continual-learning path keeps its own alpha-balanced focal weight, now as constants pinned to the removed inputs' shipped defaults, so its behaviour is unchanged. It legitimately needs its own correction: ApplyLogitAdjustment() only runs inside a training run, so a deployed model that was reloaded carries no logit offsets and would otherwise stream 31:1 data into itself uncorrected. The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a double fed to a %d conversion and had always emitted a literal 0; the |MR: segment is written as the constant its shipped defaults produced. Dropping either would have re-keyed every model and forced a from-scratch retrain of the one topology currently converged and trading. Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable "neutralized by prior correction" diagnostic. Both builds compile 0 errors, 0 warnings. No retrain forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
// EVERY BAR IS QUEUED EXACTLY ONCE. Class imbalance is corrected analytically inside
// the gradient by the logit-adjusted loss, not by duplicating minority bars here.
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//
refactor(ai): nine class-imbalance inputs down to two The imbalance section offered nine controls for one job. Audited against the code, five of them did not do what their names said at the shipped defaults: AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns whenever the adjusted loss is on, which is default. OversampleParity DEAD in training - Training.mqh gated the replay loop on !useLogitAdjustedLoss (correctly, citing Buda et al. 2018). Live only in the online-learning path. EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma damper - "replay minority bars through pass-2 oversampling" was a focal-loss switch. ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25. UseStaticPrior An exact duplicate of FreezePriorCalibration - the two were OR'd together in the single place either is read. So they were not five mechanisms fighting; they were one mechanism plus eight knobs that mostly described machinery that no longer ran. That is worse than a real conflict, because the log agreed with the names: the label-cache line printed "reps up to 28x (90% parity) (seeding era 0's class-balance oversampling)" on every run, describing an oversampling pass that had been switched off. It is fixed here too - it cost this session a wrong diagnosis. The one genuine redundancy was focal loss, running at gamma*0.125 alongside the adjusted loss: two corrections on the same axis, the exact stacking failure this file already cited Buda et al. for in two other places, damped by a replay flag whose replay path was itself dead. Removed rather than re-tuned. The plateau ladder is unaffected - its escape is the learning-rate warm restart; the gamma anneal beside it only ever stepped toward zero. WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze: LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted- Loss boolean, since a strength dial where 0 already means off does not need an on/off switch beside it. FreezePriorCalibration unchanged. It is the only one of the six corrections with a consistency guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection already ranks on - so the loss and the deploy decision optimize one thing. The online continual-learning path keeps its own alpha-balanced focal weight, now as constants pinned to the removed inputs' shipped defaults, so its behaviour is unchanged. It legitimately needs its own correction: ApplyLogitAdjustment() only runs inside a training run, so a deployed model that was reloaded carries no logit offsets and would otherwise stream 31:1 data into itself uncorrected. The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a double fed to a %d conversion and had always emitted a literal 0; the |MR: segment is written as the constant its shipped defaults produced. Dropping either would have re-keyed every model and forced a from-scratch retrain of the one topology currently converged and trading. Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable "neutralized by prior correction" diagnostic. Both builds compile 0 errors, 0 warnings. No retrain forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
// The history is worth keeping, because it is why the data-level approach was
// abandoned rather than merely re-tuned. Four successive versions of oversampling all
// collapsed, in both directions:
// v1 uncapped replication x an independent loss weight (up to ~4.5x total) ->
// Buy-only collapse, OOS ~10%, IS error 0.37->0.57 in 4 eras.
// v2 capped the ratio before splitting it between the two -> mathematically the
// same total correction as pure loss weighting, which had already failed.
// v3 replication alone, capped at 3x against a ~5.3x imbalance -> Neutral collapse,
// Buy/Sell recall 0% for 6 straight eras (2026-07-18).
// v4 replication to ~90% parity (up to 28x) -> measured across six topologies on
// 2026-07-29, every model drove ONE direction to ~50% recall and abandoned the
// other, and which direction was arbitrary. One era in 1,301 cleared the floor.
// The through-line: replication makes Buy and Sell compete for the same replicated
// capacity, and Adam's mt/sqrt(vt) normalisation (Kingma & Ba 2015) is near-invariant
// to the gradient rescaling that the loss-weighted variants relied on. Per Buda, Maki
// & Mazurowski 2018, stacking data-level and cost-level corrections on one axis is not
// reliably additive - and the logit-adjusted loss replaces BOTH with a single
// correction that is provably consistent for balanced error.
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//
refactor(ai): nine class-imbalance inputs down to two The imbalance section offered nine controls for one job. Audited against the code, five of them did not do what their names said at the shipped defaults: AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns whenever the adjusted loss is on, which is default. OversampleParity DEAD in training - Training.mqh gated the replay loop on !useLogitAdjustedLoss (correctly, citing Buda et al. 2018). Live only in the online-learning path. EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma damper - "replay minority bars through pass-2 oversampling" was a focal-loss switch. ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25. UseStaticPrior An exact duplicate of FreezePriorCalibration - the two were OR'd together in the single place either is read. So they were not five mechanisms fighting; they were one mechanism plus eight knobs that mostly described machinery that no longer ran. That is worse than a real conflict, because the log agreed with the names: the label-cache line printed "reps up to 28x (90% parity) (seeding era 0's class-balance oversampling)" on every run, describing an oversampling pass that had been switched off. It is fixed here too - it cost this session a wrong diagnosis. The one genuine redundancy was focal loss, running at gamma*0.125 alongside the adjusted loss: two corrections on the same axis, the exact stacking failure this file already cited Buda et al. for in two other places, damped by a replay flag whose replay path was itself dead. Removed rather than re-tuned. The plateau ladder is unaffected - its escape is the learning-rate warm restart; the gamma anneal beside it only ever stepped toward zero. WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze: LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted- Loss boolean, since a strength dial where 0 already means off does not need an on/off switch beside it. FreezePriorCalibration unchanged. It is the only one of the six corrections with a consistency guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection already ranks on - so the loss and the deploy decision optimize one thing. The online continual-learning path keeps its own alpha-balanced focal weight, now as constants pinned to the removed inputs' shipped defaults, so its behaviour is unchanged. It legitimately needs its own correction: ApplyLogitAdjustment() only runs inside a training run, so a deployed model that was reloaded carries no logit offsets and would otherwise stream 31:1 data into itself uncorrected. The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a double fed to a %d conversion and had always emitted a literal 0; the |MR: segment is written as the constant its shipped defaults produced. Dropping either would have re-keyed every model and forced a from-scratch retrain of the one topology currently converged and trading. Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable "neutralized by prior correction" diagnostic. Both builds compile 0 errors, 0 warnings. No retrain forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
// Pass 2 still Fisher-Yates shuffles the queue: chronological order correlates
// consecutive gradients, which is the same correlated-momentum overshoot documented at
// AI\Network.mqh's MAX_WEIGHT_DELTA comment. That reason is independent of replication
// and survives it.
//--- MINORITY REPLAY REMOVED 2026-07-31. Every bar is queued exactly once; class
//--- imbalance is corrected analytically in the gradient by the logit-adjusted loss
//--- (Menon et al. 2021) instead of by duplicating rare bars in the data. Stacking the
//--- two double-counts the same imbalance - Buda et al. 2018 - and the replay branch had
//--- in fact been gated OFF for the whole shipped configuration, so this is the code
//--- catching up with the behaviour rather than a change in it. Measured 2026-07-29
//--- across six topologies, replay made Buy and Sell compete for the same replicated
//--- capacity: every model drove ONE direction to ~50% recall and abandoned the other,
//--- and which direction was arbitrary. One era in 1,301 cleared the per-class floor.
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
int repCount = 1;
double perOccurrenceScale = 1.0;
//--- grow on demand (reserve keeps this amortized-rare) - the prealloc above is an
//--- estimate, and dropping overflow would silently starve exactly the minority
//--- classes the replication exists to protect
if(m_isTrainQueueCount + repCount > ArraySize(m_isTrainQueue))
{
int newQueueSize = m_isTrainQueueCount + repCount;
ArrayResize(m_isTrainQueue, newQueueSize, 16384);
ArrayResize(m_isTrainQueueWeightScale, newQueueSize, 16384);
ArrayResize(m_isTrainQueuePrimary, newQueueSize, 16384);
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
ArrayResize(m_isTrainQueueCand, newQueueSize, 16384);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
for(int rep = 0; rep < repCount; rep++)
{
m_isTrainQueue[m_isTrainQueueCount] = i;
m_isTrainQueueWeightScale[m_isTrainQueueCount] = perOccurrenceScale;
//--- rep 0 is this bar's single "counts once" occurrence - see m_isTrainQueuePrimary.
//--- Every rep still trains; only the reported IS accuracy looks at this flag.
m_isTrainQueuePrimary[m_isTrainQueueCount] = (rep == 0);
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
m_isTrainQueueCand[m_isTrainQueueCount] = -1;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_isTrainQueueCount++;
}
}
}
stop = IsStopped() || m_trainingStopRequested;
if(!stop && i > 0 && GetTickCount() - chunkStartTick >= TRAIN_TIME_BUDGET_MS)
{
//--- yield: save exactly enough to resume this same era, mid-bar-loop, on the next call -
//--- see m_trainRunActive's declaration comment for why this must happen instead of
//--- letting one era (or the whole run) process synchronously to completion
m_resumeBars = bars;
m_resumeTotalIter = totalIter;
m_resumeOosCutoff = oosCutoff;
m_resumeAddLoop = add_loop;
m_resumeBarIndex = i - 1;
m_eraResumePending = true;
// Save this model's own learning-rate trajectory back out of the shared global before
// yielding - see m_modelEta's declaration comment.
m_modelEta = eta;
return;
}
}
diag: an era that discards itself now says so instead of scanning forever add_loop is exactly "at least one bar produced a usable feature window". When it stays false, pass 2, pass 3, the era counter, the checkpoint and every log line in the era-end block are ALL skipped: Train() returns having done nothing, m_eraResumePending is still false, and the next call restarts the SAME era from bar 0. That is an infinite 0->100% "scan" loop that prints absolutely nothing - the only remaining silent restart path in Train(), and it matches the reported symptom exactly. Pass 1 now counts usable vs unusable windows and reports at the pass boundary, which demonstrably executes: - total failure routes through ReportTrainStall (already capped at one line a minute, and carries the run-state flags) naming the counts, the required window width and the bar count - success prints how long the scan took and how many samples it handed to pass 2, but only once the era has passed 10s - a fast era stays as quiet as before, a slow one distinguishes "advancing" from "sweeping the same bars forever" A PARTIAL failure is normal and deliberately does not shout: pass 1 walks oldest-to-newest and the deepest bars predate the indicators' warm-up, so those windows fail and are cached as misses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:29:33 -04:00
//--- PASS 1 IS OVER (the yield above is the only other way out of that loop). This is the point
//--- that decides whether the era does any work at all, and until now it said nothing.
//--- add_loop is exactly "m_passWindowOk > 0". When it is false, pass 2, pass 3, the era counter,
//--- the checkpoint and every log line below are ALL skipped - Train() returns having done
//--- nothing, m_eraResumePending is still false, and the next call restarts the SAME era from
//--- bar 0. An infinite, completely silent 0->100% "scan" loop with no journal output whatsoever,
//--- which is what the panel showed on 2026-08-10 once the dispatch fix let pass 1 run at speed.
//--- A PARTIAL failure is normal and must not be alarming: the loop walks oldest-to-newest and
//--- the deepest bars legitimately predate the indicators' warm-up, so those windows fail and are
//--- cached as misses. Only a TOTAL failure is a defect, so only that one shouts.
if(!stop)
{
if(!add_loop)
fix: a resumed model cached a cold ATR as permanent, so it never trained BufferTempData cached EVERY failure - m_featureCacheHasValue[idx]=true with m_featureCacheValid[idx]=false - and the cache never re-tries a miss. So a single feature read taken before the terminal had finished calculating the indicator buffers marked those bars unusable for the rest of the process, even though the data arrived milliseconds later. MT5 fills an indicator's buffers asynchronously after the handle is created, and a cold ATR returns 0 for EVERY index, not just its warm-up tail. BufferTempDataCompute rejects a bar with no ATR (correctly - the price features would be meaningless), so the whole window failed, and the whole cache was poisoned. Only resumed models were hit, because only they read features that early. Topology.mqh sets m_warmupPassesRemaining = netLoaded ? 0 : 3: a fresh start sits through three separately-scheduled Train() calls before anything touches a feature, which is exactly what those passes are for. A resumed one skips them and TuneIndicatorsAndTrain drives StartLabelCachePrebuild and the MI report from the first chart event. Its rationale - "a restart already has a proven-synced history" - holds for HISTORY and not for INDICATORS, which are recreated every process start. Downstream: BuildFeatureWindow failed on every bar of every era, so add_loop never went true, so pass 2, pass 3, the era counter and the checkpoint were all skipped and pass 1 swept 0->100% forever. The "0 samples" MI report line at startup was the same failure, four seconds earlier, already visible in the log. - a miss is now cached only when it is PERMANENT; the two "not ready yet" guards mark m_featureFailTransient and are recomputed on the next visit. Steady-state cost is ~ind_Periods bars per era, not 54k. - an era that discards itself now drops the feature cache before restarting, so any remaining cause of this state self-heals instead of looping. Deleting the .nnw "fixed" this only by turning the model back into a fresh one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:41:31 -04:00
{
//--- SELF-HEAL BEFORE RESTARTING. A sweep that produced no usable window at all will
//--- produce exactly the same result next time unless something changes, because every bar
//--- it touched is now answered from the feature cache. Dropping the cached verdicts forces
//--- the next sweep to recompute against whatever the terminal has finished loading since -
//--- which is the difference between recovering a few seconds later and looping forever.
//--- The known cause of this state (a cold ATR read by a resumed model before its
//--- indicators had calculated) is fixed at source in BufferTempData, so reaching here at
//--- all now means an unknown cause; recover anyway rather than spin, and say so.
ArrayInitialize(m_featureCacheHasValue, false);
diag: an era that discards itself now says so instead of scanning forever add_loop is exactly "at least one bar produced a usable feature window". When it stays false, pass 2, pass 3, the era counter, the checkpoint and every log line in the era-end block are ALL skipped: Train() returns having done nothing, m_eraResumePending is still false, and the next call restarts the SAME era from bar 0. That is an infinite 0->100% "scan" loop that prints absolutely nothing - the only remaining silent restart path in Train(), and it matches the reported symptom exactly. Pass 1 now counts usable vs unusable windows and reports at the pass boundary, which demonstrably executes: - total failure routes through ReportTrainStall (already capped at one line a minute, and carries the run-state flags) naming the counts, the required window width and the bar count - success prints how long the scan took and how many samples it handed to pass 2, but only once the era has passed 10s - a fast era stays as quiet as before, a slow one distinguishes "advancing" from "sweeping the same bars forever" A PARTIAL failure is normal and deliberately does not shout: pass 1 walks oldest-to-newest and the deepest bars predate the indicators' warm-up, so those windows fail and are cached as misses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:29:33 -04:00
//--- Routed through ReportTrainStall rather than printed directly: a discarded era restarts
//--- immediately, so this condition repeats as fast as pass 1 can sweep, and an unthrottled
//--- line would bury the journal. That helper already rate-limits to one line a minute and
//--- carries the run-state flags this needs read alongside it.
diag: name the cause when every feature window fails, and enforce the width contract Era 0 stalls with "NOT ONE of 54681 scanned bars produced a usable feature window, windows ok=0 failed=54681" and nothing else. That line reads identically for a cold ATR, a conditionally-missing optional feature block and an out-of-range index, so it cannot be diagnosed without one restart per hypothesis. Two changes: 1. WIDTH CONTRACT in BufferTempData. Every enabled block must emit exactly m_neuronsCount values on EVERY bar. A block that emits its values on some bars and skips them on others (indicator, panel or series unavailable for that bar) does not merely shorten the window - it SHIFTS every feature after it into the wrong slot, and the net then trains on silently misaligned inputs that still look like a valid window to everything downstream. Now rejected, rolled back and reported once, naming the optional blocks (XA / SPR / swing context) as the ones carrying an availability test. Worth having independently of the current stall. 2. BuildFeatureWindow records WHICH lookback slot rejected and how much of the window was assembled, and the pass-1 stall report renders it: "slot 0 of 20 REJECTED (window had 0 of 760)" is an indicator warm-up or history-edge read; "every lookback bar ACCEPTED and the window was still short: 640 of 760" is a missing 6-value block. No behaviour change on a healthy run: the width check is an equality that already holds, and the diagnostics render only inside the total-failure branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 10:31:56 -04:00
//--- The CAUSE, not just the count. "0 of 54681 usable" reads identically for a cold ATR,
//--- a conditionally-missing optional feature block and an out-of-range index, and telling
//--- them apart by reasoning cost a whole debugging session once already.
string whyLine;
if(m_windowFailSlot == -2)
whyLine = "no window has been attempted yet this run (m_windowFailSlot unset) - the"
" failure is upstream of BuildFeatureWindow";
else
if(m_windowFailSlot < 0)
whyLine = StringFormat("every lookback bar was ACCEPTED and the window was still"
" short: %d of %d values. A feature block emitted fewer values"
" than m_neuronsCount promises", m_windowFailTotal,
(int)m_historyBars * m_neuronsCount);
else
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
//--- THE BLOCK, NOT JUST THE SLOT. The old form ended at "slot 0 REJECTED (window had
//--- 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"),
//--- and that guess was read as a finding five times across 2026-08-17. m_featureFailBlock
//--- is written by the guard that actually returned false, and IndicatorDepthReport()
//--- prints every handle's BarsCalculated() beside it, so the next occurrence is READ
//--- rather than reasoned about. A total failure (ok=0) is itself evidence: it means the
//--- newest anchors failed too, which no depth shortfall can cause.
{
string byBlock = m_featureFailBlock;
if(byBlock == "")
byBlock = "(no guard recorded - the rejection came from a TempData.Add failure,"
" not a data guard)";
whyLine = StringFormat("lookback slot %d of %d REJECTED the bar at series index %d"
" (window had %d of %d values). REJECTED BY: %s. Slot 0 is the"
" DEEPEST lookback of the window, so with ok=0 the newest anchors"
" failed as well - which rules out a plain history-edge read and"
" points at a buffer that is unreadable at every index."
" Per-indicator depth:%s",
m_windowFailSlot, (int)m_historyBars, m_featureFailIdx,
m_windowFailTotal, (int)m_historyBars * m_neuronsCount,
byBlock, IndicatorDepthReport());
}
diag: an era that discards itself now says so instead of scanning forever add_loop is exactly "at least one bar produced a usable feature window". When it stays false, pass 2, pass 3, the era counter, the checkpoint and every log line in the era-end block are ALL skipped: Train() returns having done nothing, m_eraResumePending is still false, and the next call restarts the SAME era from bar 0. That is an infinite 0->100% "scan" loop that prints absolutely nothing - the only remaining silent restart path in Train(), and it matches the reported symptom exactly. Pass 1 now counts usable vs unusable windows and reports at the pass boundary, which demonstrably executes: - total failure routes through ReportTrainStall (already capped at one line a minute, and carries the run-state flags) naming the counts, the required window width and the bar count - success prints how long the scan took and how many samples it handed to pass 2, but only once the era has passed 10s - a fast era stays as quiet as before, a slow one distinguishes "advancing" from "sweeping the same bars forever" A PARTIAL failure is normal and deliberately does not shout: pass 1 walks oldest-to-newest and the deepest bars predate the indicators' warm-up, so those windows fail and are cached as misses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:29:33 -04:00
ReportTrainStall(StringFormat("pass 1 finished but NOT ONE of %d scanned bars produced a"
" usable feature window, so the era is discarded and restarts"
fix: a resumed model cached a cold ATR as permanent, so it never trained BufferTempData cached EVERY failure - m_featureCacheHasValue[idx]=true with m_featureCacheValid[idx]=false - and the cache never re-tries a miss. So a single feature read taken before the terminal had finished calculating the indicator buffers marked those bars unusable for the rest of the process, even though the data arrived milliseconds later. MT5 fills an indicator's buffers asynchronously after the handle is created, and a cold ATR returns 0 for EVERY index, not just its warm-up tail. BufferTempDataCompute rejects a bar with no ATR (correctly - the price features would be meaningless), so the whole window failed, and the whole cache was poisoned. Only resumed models were hit, because only they read features that early. Topology.mqh sets m_warmupPassesRemaining = netLoaded ? 0 : 3: a fresh start sits through three separately-scheduled Train() calls before anything touches a feature, which is exactly what those passes are for. A resumed one skips them and TuneIndicatorsAndTrain drives StartLabelCachePrebuild and the MI report from the first chart event. Its rationale - "a restart already has a proven-synced history" - holds for HISTORY and not for INDICATORS, which are recreated every process start. Downstream: BuildFeatureWindow failed on every bar of every era, so add_loop never went true, so pass 2, pass 3, the era counter and the checkpoint were all skipped and pass 1 swept 0->100% forever. The "0 samples" MI report line at startup was the same failure, four seconds earlier, already visible in the log. - a miss is now cached only when it is PERMANENT; the two "not ready yet" guards mark m_featureFailTransient and are recomputed on the next visit. Steady-state cost is ~ind_Periods bars per era, not 54k. - an era that discards itself now drops the feature cache before restarting, so any remaining cause of this state self-heals instead of looping. Deleting the .nnw "fixed" this only by turning the model back into a fresh one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:41:31 -04:00
" from scratch (feature cache dropped so the next sweep"
" recomputes) - windows ok=%d failed=%d, BuildFeatureWindow"
diag: an era that discards itself now says so instead of scanning forever add_loop is exactly "at least one bar produced a usable feature window". When it stays false, pass 2, pass 3, the era counter, the checkpoint and every log line in the era-end block are ALL skipped: Train() returns having done nothing, m_eraResumePending is still false, and the next call restarts the SAME era from bar 0. That is an infinite 0->100% "scan" loop that prints absolutely nothing - the only remaining silent restart path in Train(), and it matches the reported symptom exactly. Pass 1 now counts usable vs unusable windows and reports at the pass boundary, which demonstrably executes: - total failure routes through ReportTrainStall (already capped at one line a minute, and carries the run-state flags) naming the counts, the required window width and the bar count - success prints how long the scan took and how many samples it handed to pass 2, but only once the era has passed 10s - a fast era stays as quiet as before, a slow one distinguishes "advancing" from "sweeping the same bars forever" A PARTIAL failure is normal and deliberately does not shout: pass 1 walks oldest-to-newest and the deepest bars predate the indicators' warm-up, so those windows fail and are cached as misses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:29:33 -04:00
" needs %d values per bar (historyBars=%d x featuresPerBar=%d)"
diag: name the cause when every feature window fails, and enforce the width contract Era 0 stalls with "NOT ONE of 54681 scanned bars produced a usable feature window, windows ok=0 failed=54681" and nothing else. That line reads identically for a cold ATR, a conditionally-missing optional feature block and an out-of-range index, so it cannot be diagnosed without one restart per hypothesis. Two changes: 1. WIDTH CONTRACT in BufferTempData. Every enabled block must emit exactly m_neuronsCount values on EVERY bar. A block that emits its values on some bars and skips them on others (indicator, panel or series unavailable for that bar) does not merely shorten the window - it SHIFTS every feature after it into the wrong slot, and the net then trains on silently misaligned inputs that still look like a valid window to everything downstream. Now rejected, rolled back and reported once, naming the optional blocks (XA / SPR / swing context) as the ones carrying an availability test. Worth having independently of the current stall. 2. BuildFeatureWindow records WHICH lookback slot rejected and how much of the window was assembled, and the pass-1 stall report renders it: "slot 0 of 20 REJECTED (window had 0 of 760)" is an indicator warm-up or history-edge read; "every lookback bar ACCEPTED and the window was still short: 640 of 760" is a missing 6-value block. No behaviour change on a healthy run: the width check is an equality that already holds, and the diagnostics render only inside the total-failure branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 10:31:56 -04:00
" over %d bars | LAST FAILURE: %s",
diag: an era that discards itself now says so instead of scanning forever add_loop is exactly "at least one bar produced a usable feature window". When it stays false, pass 2, pass 3, the era counter, the checkpoint and every log line in the era-end block are ALL skipped: Train() returns having done nothing, m_eraResumePending is still false, and the next call restarts the SAME era from bar 0. That is an infinite 0->100% "scan" loop that prints absolutely nothing - the only remaining silent restart path in Train(), and it matches the reported symptom exactly. Pass 1 now counts usable vs unusable windows and reports at the pass boundary, which demonstrably executes: - total failure routes through ReportTrainStall (already capped at one line a minute, and carries the run-state flags) naming the counts, the required window width and the bar count - success prints how long the scan took and how many samples it handed to pass 2, but only once the era has passed 10s - a fast era stays as quiet as before, a slow one distinguishes "advancing" from "sweeping the same bars forever" A PARTIAL failure is normal and deliberately does not shout: pass 1 walks oldest-to-newest and the deepest bars predate the indicators' warm-up, so those windows fail and are cached as misses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:29:33 -04:00
totalIter, m_passWindowOk, m_passWindowFail,
(int)m_historyBars * m_neuronsCount,
diag: name the cause when every feature window fails, and enforce the width contract Era 0 stalls with "NOT ONE of 54681 scanned bars produced a usable feature window, windows ok=0 failed=54681" and nothing else. That line reads identically for a cold ATR, a conditionally-missing optional feature block and an out-of-range index, so it cannot be diagnosed without one restart per hypothesis. Two changes: 1. WIDTH CONTRACT in BufferTempData. Every enabled block must emit exactly m_neuronsCount values on EVERY bar. A block that emits its values on some bars and skips them on others (indicator, panel or series unavailable for that bar) does not merely shorten the window - it SHIFTS every feature after it into the wrong slot, and the net then trains on silently misaligned inputs that still look like a valid window to everything downstream. Now rejected, rolled back and reported once, naming the optional blocks (XA / SPR / swing context) as the ones carrying an availability test. Worth having independently of the current stall. 2. BuildFeatureWindow records WHICH lookback slot rejected and how much of the window was assembled, and the pass-1 stall report renders it: "slot 0 of 20 REJECTED (window had 0 of 760)" is an indicator warm-up or history-edge read; "every lookback bar ACCEPTED and the window was still short: 640 of 760" is a missing 6-value block. No behaviour change on a healthy run: the width check is an equality that already holds, and the diagnostics render only inside the total-failure branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 10:31:56 -04:00
(int)m_historyBars, m_neuronsCount, bars, whyLine));
fix(resume): model reload stalled training - three hardenings on the resume path A resumed META model hot-looped pass 1 (0->100% scan oscillation, silent for 3 minutes until the stall reporter fired) because EVERY window failed at the first AD/Wyckoff feature: the init-time param adoption called ReInitADIndicators unconditionally, destroying five freshly-calculating indicator instances to recreate them with BYTE-IDENTICAL params (verified by parsing the .nnw header - the MI tuner had kept the configured settings), at process start, on a box with 1 GB free of 31. The replacements sat cold for 6+ minutes while full-history resweeps starved the indicator threads harder. - AdoptIndicatorParams: installs a loaded param set into the tuner and rebuilds handles ONLY when the set actually differs from what the live indicators run. Both call sites (resume init + panel reload) use it. - Resumed models get the same 3 warm-up passes as fresh ones. The skip was the shared root cause of the cold-ATR (ba13eef), cold-AD (2026-08-11) and this incident - custom indicators recompute from scratch every process start regardless of what the .nnw proves. - Cold-sweep backoff: a pass-1 sweep in which every window failed on a TRANSIENT cause arms a 5s era-start pause instead of an immediate full-history resweep, so the retry loop stops consuming the CPU/memory the warming indicators need. The stall reporter names the backoff branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 10:23:11 -04:00
//--- transient cause (cold indicator) -> arm the era-start backoff instead of resweeping
//--- at full speed; see the backoff block at the top of the fresh-era branch.
fix(altdata): median-fill instead of zero-fill, and a one-shot feature-vector autopsy ALT-DATA AUDIT. The files themselves are healthy - all six symbols, 6,073 daily rows, 2010-01-01 to 2026-08-17, no constant or degenerate columns, sane tails (mac_cpi/mac_unemp flat ~47d is monthly data behaving correctly). The problem is not the data, it is what happens where the data ISN'T. CAltDataPanel::Features() returned an all-ZERO vector for any bar older than the file's first row, and left blank cells at 0 too. Both were deliberate ('the block is additive context and must degrade, never reject the bar') and that reasoning holds for the CHANGE columns - but half these features are LEVELS: vix, ivol, mac_y10, mac_cpi, mac_unemp, eia_util. For a level, 0 is not a missing reading, it is an impossible one far outside the series' range. VIX does not visit zero. And the spike lands in exactly the wrong place. Every alt file starts 2010-01-01 while the charts run far deeper - USDJPY H4 reaches ~1994, roughly HALF its history - so 'alt block is all zeros' is precisely the predicate 'this bar is older than 2010'. The IS/OOS split is chronological, so that predicate covers ~half of IS and none of OOS: an in-sample feature guaranteed to be useless out-of-sample, and a bimodal input for the first BatchNorm to normalise. Not a lookahead leak - a distribution corruption, which is quieter and was never reported anywhere. Now filled with the column MEDIAN over the covered range. A constant cannot leak whatever its source - it takes the same value on every pre-coverage bar, so it carries no information about which of those bars won - which is what makes a median computed over later data legitimate here. Median not mean because the series are skewed. Blank cells get the same treatment (eia_stk_idx1y alone has 181 blanks in 6,073 rows) and the count is now logged at load. THE BACKOFF WAS ALREADY THERE AND WAS DEAD. Training.mqh arms m_coldSweepTick on m_featureFailTransient, but only the open/ATR guards ever set that flag, so f0cf659's cold ADMovingAverage looked PERMANENT and the sweep re-ran at full speed forever. Setting the flag in the indicator guards revives the mechanism that was already designed for this; no second backoff was needed and the one I first wrote has been removed in favour of it. SELF-HEALING, as asked. ReportFeatureHealth() runs once, the first time pass 1 produces usable windows, samples ~400 bars spread across the whole training range and names every feature slot that is CONSTANT or mostly-zero, tagging alt-block slots as alt[i]. Both of today's failures were the same shape - a block silently produces nothing while every downstream number stays plausible - and neither an accuracy figure nor a model can tell 'this feature is always 0' from 'this feature is genuinely 0 here'. Evenly spaced sampling so a block that dies only in deep history is caught as surely as one dead everywhere. A report, not a gate: a rare-flag feature can be legitimately constant, and refusing to train would turn a diagnostic into an outage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 12:53:51 -04:00
//--- THIS BACKOFF WAS DEAD UNTIL 2026-08-17 and that is why the USDJPY/XAUUSD stall never
//--- recovered. It arms only on m_featureFailTransient, and of the guards that can reject
//--- a bar only the open/ATR pair ever set that flag - the MA, RSI, MACD and Ichimoku
//--- guards did not. A cold ADMovingAverage therefore looked PERMANENT, so the sweep was
//--- re-run at full speed forever, and six instances doing that on a six-core box starved
//--- the very indicator they were waiting on. The mechanism was right; nothing reached it.
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
//---
//--- 2026-08-17 (second pass): the backoff is now UNCONDITIONAL on a total failure, not
//--- gated on m_featureFailTransient. Naming every guard's flag correctly is a list that has
//--- to stay correct forever - the same shape of fix the feature cache abandoned above for
//--- the same reason - and being wrong once costs a chart. The gate is also pointless here:
//--- whether the cause is transient or permanent, a sweep in which ZERO of 50,163 bars
//--- produced a window will produce zero again if it restarts a millisecond later, and doing
//--- so at full speed is what starved six indicator threads on a six-core box. Back off in
//--- both cases. The flag is kept for what it legitimately decides - whether the miss may be
//--- cached (see BufferTempData) - which is a per-bar question, not a scheduling one.
m_coldSweepTick = GetTickCount();
fix: a resumed model cached a cold ATR as permanent, so it never trained BufferTempData cached EVERY failure - m_featureCacheHasValue[idx]=true with m_featureCacheValid[idx]=false - and the cache never re-tries a miss. So a single feature read taken before the terminal had finished calculating the indicator buffers marked those bars unusable for the rest of the process, even though the data arrived milliseconds later. MT5 fills an indicator's buffers asynchronously after the handle is created, and a cold ATR returns 0 for EVERY index, not just its warm-up tail. BufferTempDataCompute rejects a bar with no ATR (correctly - the price features would be meaningless), so the whole window failed, and the whole cache was poisoned. Only resumed models were hit, because only they read features that early. Topology.mqh sets m_warmupPassesRemaining = netLoaded ? 0 : 3: a fresh start sits through three separately-scheduled Train() calls before anything touches a feature, which is exactly what those passes are for. A resumed one skips them and TuneIndicatorsAndTrain drives StartLabelCachePrebuild and the MI report from the first chart event. Its rationale - "a restart already has a proven-synced history" - holds for HISTORY and not for INDICATORS, which are recreated every process start. Downstream: BuildFeatureWindow failed on every bar of every era, so add_loop never went true, so pass 2, pass 3, the era counter and the checkpoint were all skipped and pass 1 swept 0->100% forever. The "0 samples" MI report line at startup was the same failure, four seconds earlier, already visible in the log. - a miss is now cached only when it is PERMANENT; the two "not ready yet" guards mark m_featureFailTransient and are recomputed on the next visit. Steady-state cost is ~ind_Periods bars per era, not 54k. - an era that discards itself now drops the feature cache before restarting, so any remaining cause of this state self-heals instead of looping. Deleting the .nnw "fixed" this only by turning the model back into a fresh one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:41:31 -04:00
}
diag: an era that discards itself now says so instead of scanning forever add_loop is exactly "at least one bar produced a usable feature window". When it stays false, pass 2, pass 3, the era counter, the checkpoint and every log line in the era-end block are ALL skipped: Train() returns having done nothing, m_eraResumePending is still false, and the next call restarts the SAME era from bar 0. That is an infinite 0->100% "scan" loop that prints absolutely nothing - the only remaining silent restart path in Train(), and it matches the reported symptom exactly. Pass 1 now counts usable vs unusable windows and reports at the pass boundary, which demonstrably executes: - total failure routes through ReportTrainStall (already capped at one line a minute, and carries the run-state flags) naming the counts, the required window width and the bar count - success prints how long the scan took and how many samples it handed to pass 2, but only once the era has passed 10s - a fast era stays as quiet as before, a slow one distinguishes "advancing" from "sweeping the same bars forever" A PARTIAL failure is normal and deliberately does not shout: pass 1 walks oldest-to-newest and the deepest bars predate the indicators' warm-up, so those windows fail and are cached as misses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:29:33 -04:00
else
{
//--- Healthy pass 1. Quiet on a fast era, but an era that has already taken longer than
//--- PASS1_LOUD_AFTER_MS is one somebody is watching a progress bar on, and the single most
//--- useful thing to tell them is that the scan ENDED and what it handed to pass 2 - that
//--- is what separates "slow but advancing" from "sweeping the same bars forever".
fix(altdata): median-fill instead of zero-fill, and a one-shot feature-vector autopsy ALT-DATA AUDIT. The files themselves are healthy - all six symbols, 6,073 daily rows, 2010-01-01 to 2026-08-17, no constant or degenerate columns, sane tails (mac_cpi/mac_unemp flat ~47d is monthly data behaving correctly). The problem is not the data, it is what happens where the data ISN'T. CAltDataPanel::Features() returned an all-ZERO vector for any bar older than the file's first row, and left blank cells at 0 too. Both were deliberate ('the block is additive context and must degrade, never reject the bar') and that reasoning holds for the CHANGE columns - but half these features are LEVELS: vix, ivol, mac_y10, mac_cpi, mac_unemp, eia_util. For a level, 0 is not a missing reading, it is an impossible one far outside the series' range. VIX does not visit zero. And the spike lands in exactly the wrong place. Every alt file starts 2010-01-01 while the charts run far deeper - USDJPY H4 reaches ~1994, roughly HALF its history - so 'alt block is all zeros' is precisely the predicate 'this bar is older than 2010'. The IS/OOS split is chronological, so that predicate covers ~half of IS and none of OOS: an in-sample feature guaranteed to be useless out-of-sample, and a bimodal input for the first BatchNorm to normalise. Not a lookahead leak - a distribution corruption, which is quieter and was never reported anywhere. Now filled with the column MEDIAN over the covered range. A constant cannot leak whatever its source - it takes the same value on every pre-coverage bar, so it carries no information about which of those bars won - which is what makes a median computed over later data legitimate here. Median not mean because the series are skewed. Blank cells get the same treatment (eia_stk_idx1y alone has 181 blanks in 6,073 rows) and the count is now logged at load. THE BACKOFF WAS ALREADY THERE AND WAS DEAD. Training.mqh arms m_coldSweepTick on m_featureFailTransient, but only the open/ATR guards ever set that flag, so f0cf659's cold ADMovingAverage looked PERMANENT and the sweep re-ran at full speed forever. Setting the flag in the indicator guards revives the mechanism that was already designed for this; no second backoff was needed and the one I first wrote has been removed in favour of it. SELF-HEALING, as asked. ReportFeatureHealth() runs once, the first time pass 1 produces usable windows, samples ~400 bars spread across the whole training range and names every feature slot that is CONSTANT or mostly-zero, tagging alt-block slots as alt[i]. Both of today's failures were the same shape - a block silently produces nothing while every downstream number stays plausible - and neither an accuracy figure nor a model can tell 'this feature is always 0' from 'this feature is genuinely 0 here'. Evenly spaced sampling so a block that dies only in deep history is caught as surely as one dead everywhere. A report, not a gate: a rare-flag feature can be legitimately constant, and refusing to train would turn a diagnostic into an outage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 12:53:51 -04:00
//--- FIRST HEALTHY SWEEP is the only moment the assembled feature vector is known to be
//--- readable and not yet been trained on - so it is where the block-level autopsy belongs.
//--- See ReportFeatureHealth() for the two silent failures that motivated it.
ReportFeatureHealth(bars);
feat(gates): derive the recall floor instead of configuring it, publish what the configuration can PROVE, and stop an arbitrary member driving live exits Three changes, all from the same principle: measure what is there before aiming at it, and never certify a number you do not trade. 1. THE RECALL FLOOR IS DERIVED, AND IT MOVES BELOW CHANCE. MinRecall=40 was a constant doing a statistical job. Its reference point is the 33.3% recall a zero-skill 3-class model gets on EVERY class, and against that the constant was accidentally calibrated for exactly one sample size: on USDJPY CONV (n_eff 195) 40% is chance + 2.0 SE; on SP500 PAI (n_eff 42) the same 40% is chance + 0.9 SE. One chart was being held to a bar twice as strict as the other, for no reason anyone chose. CollapseRecallFloorPct() computes it per class from that class's own effective sample - EffectiveSampleSize(), so the overlap deflation the rest of the gates use applies here too - as chance - EDGE_MIN_SIGMAS x SE. 26.5% at n_eff 195, 18.8% at n_eff 42. BELOW chance, deliberately, and this is the substantive change rather than the arithmetic. This gate's only job is refusing to call a COLLAPSED model converged. It is not a quality bar; the deploy gate is the quality bar and it is already rigorous (chance + 2 SE on the deflated sample, Sidak over candidate eras, then the cross-instrument pooled certificate). A convergence gate that ALSO demands provably-above-chance recall on all three classes double-counts that job, and it has failed that way twice here: MinRecall=60 blocked every SP500 H1 run in 2026-07, and the 40 that replaced it made Neutral structurally unreachable once first-touch resolution cut Neutral to a 0.65% residue. A floor nothing can reach does not make a funded account safer, it stops the run converging at all. Testing significantly BELOW chance instead catches what a fixed 40 was actually catching - a model that has stopped emitting a class - and cannot become unreachable by construction. It also fixes the direction the old constant scaled: it now widens on a thin OOS window, where low recall genuinely cannot be told from noise, and tightens on a rich one. Today's SP500 PAI (Buy 51 / Sell 18 / Neutral 30) is still correctly blocked on Sell. This also resolves a standing contradiction the code half-admitted at the isBetterEra comment: selection ranks on coverage-weighted PRECISION while convergence gated on RECALL, so a sparse high-precision abstainer - precisely the model that could clear the deploy bar - was blocked by the floor. The era line now PRINTS the derived floor. Anyone comparing these recalls against a remembered "40" is reading the wrong bar. 2. DETECTABILITY: WHAT THIS CONFIGURATION COULD PROVE, BEFORE IT TRAINS. The DEPLOY BAR line states the bar. It never said what reaching it would take, and that is the actionable direction. ReportDetectability() inverts the same identity - the gate passes when edge >= z x sqrt(p(1-p)/n_eff), so certifying an edge d needs n_eff >= z^2 p(1-p)/d^2 independent calls, hence L times as many raw ones - and prints a +2 / +5 / +10pp ladder as required independent calls, raw calls, and share of the OOS window, marking any rung that needs more than the window holds IMPOSSIBLE. Every term is a property of the CONFIGURATION - geometry via break-even, horizon via mean label lifespan, window via oosCutoff - so no amount of training moves any of them. It fires once, at the first healthy sweep, beside ReportFeatureHealth, for the same reason: that is the first moment the bar grid, the measured geometry and the lifespan are real numbers rather than defaults. It gates nothing. This is the EdgeFinder discipline applied to our own gate: establish what the market and the measurement design have to offer, then point the net at it - rather than spending a thousand eras chasing something this OOS window could never certify. 3. AN ARBITRARY MEMBER WAS DRIVING LIVE EXITS AND TRAILING (user-identified). Every ensemble member ran g_LiveAISignedConfidence = SignedAIConfidence(); unconditionally, every tick. Last writer wins. Its consumers are the AI early-exit route (CExpertSignalCustom::LiveSignedConfidence) and TrailingIntelligent - so on a four-model chart an LSTM entry could be closed, and its stop moved, on the Perceptron's opinion alone, decided by scheduling order. Not the vote, not a weighted blend. Now the mean across registered members, matching how the ensemble actually trades: the open decision is the weighted-average vote, and an abstaining member contributes 0 and dilutes exactly as it does there. Members still training read 0, so a half-trained ensemble reads WEAKER rather than louder - the safe direction for an exit trigger. Deployed and paused members are included, which is the opposite of the era barrier's exemption rule and correct for the opposite reason: that one asks who must be waited for, this asks who has an opinion. Latent today and staying that way for now by choice - Min_Vote_Close ships Disabled (101, unreachable on both scales it drives) and TrailingStrategy is off, so live exits are SL/TP only and the certified hold-to-barrier win rate is what actually gets traded. Fixed now precisely because the plan is to enable vote exits once the models are accurate, at which point a scheduling-order exit would be both harmful and very hard to see. STILL OPEN, and needs a decision before vote exits go on: the member gate and the ensemble vote gate both grade hold-to-barrier, so enabling vote exits makes the certified number stop describing the traded one. Warrior_EA.mq5 currently argues barrier models may keep vote exits because "their label IS the vote's own horizon" - that does not hold, since a vote flip at bar 5 of a 64-bar horizon is not the target-before-stop outcome the gate measured. Either grade the OOS call on the real exit rule (first of SL / TP / vote-flip / horizon) through the fill engine, or set HoldToBarrier for ensemble members so the policy cannot drift from the certificate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:15:08 -04:00
//--- Same moment, same reason: the first sweep that produced usable windows is the first
//--- point at which the era's bar grid, the measured barrier geometry and the label lifespan
//--- are all real numbers rather than defaults. What this one says is what the CONFIGURATION
//--- can prove - published before the run spends a thousand eras chasing something the OOS
//--- window could never certify.
//--- oosCutoff IS the OOS count, not the IS/OOS boundary counted from the other end:
//--- pass 3 grades `isOOS = (i < oosCutoff)`, so these are the bars the deploy gate will
//--- ever get to see, and they are the only ones this budget may be denominated in.
ReportDetectability(oosCutoff);
diag: an era that discards itself now says so instead of scanning forever add_loop is exactly "at least one bar produced a usable feature window". When it stays false, pass 2, pass 3, the era counter, the checkpoint and every log line in the era-end block are ALL skipped: Train() returns having done nothing, m_eraResumePending is still false, and the next call restarts the SAME era from bar 0. That is an infinite 0->100% "scan" loop that prints absolutely nothing - the only remaining silent restart path in Train(), and it matches the reported symptom exactly. Pass 1 now counts usable vs unusable windows and reports at the pass boundary, which demonstrably executes: - total failure routes through ReportTrainStall (already capped at one line a minute, and carries the run-state flags) naming the counts, the required window width and the bar count - success prints how long the scan took and how many samples it handed to pass 2, but only once the era has passed 10s - a fast era stays as quiet as before, a slow one distinguishes "advancing" from "sweeping the same bars forever" A PARTIAL failure is normal and deliberately does not shout: pass 1 walks oldest-to-newest and the deepest bars predate the indicators' warm-up, so those windows fail and are cached as misses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:29:33 -04:00
const uint PASS1_LOUD_AFTER_MS = 10000;
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- The calibration band is reported here, beside the queue count it is subtracted from, so
//--- the two are read together: a run where the band silently came out empty (see
//--- CalibBandBars) is one whose operating point is no longer being refitted at all, and the
//--- only place that is visible is next to the number it should have reduced.
diag: an era that discards itself now says so instead of scanning forever add_loop is exactly "at least one bar produced a usable feature window". When it stays false, pass 2, pass 3, the era counter, the checkpoint and every log line in the era-end block are ALL skipped: Train() returns having done nothing, m_eraResumePending is still false, and the next call restarts the SAME era from bar 0. That is an infinite 0->100% "scan" loop that prints absolutely nothing - the only remaining silent restart path in Train(), and it matches the reported symptom exactly. Pass 1 now counts usable vs unusable windows and reports at the pass boundary, which demonstrably executes: - total failure routes through ReportTrainStall (already capped at one line a minute, and carries the run-state flags) naming the counts, the required window width and the bar count - success prints how long the scan took and how many samples it handed to pass 2, but only once the era has passed 10s - a fast era stays as quiet as before, a slow one distinguishes "advancing" from "sweeping the same bars forever" A PARTIAL failure is normal and deliberately does not shout: pass 1 walks oldest-to-newest and the deepest bars predate the indicators' warm-up, so those windows fail and are cached as misses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:29:33 -04:00
string pass1Line = StringFormat("%s: era %d pass 1 done in %.0fs - %d of %d bars usable"
" (%d failed, normal over the oldest bars), %d queued for"
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
" backprop | %d bars held out to calibrate the operating"
" point (+2x%d purged around it)", ID, (int)m_eraCount,
diag: an era that discards itself now says so instead of scanning forever add_loop is exactly "at least one bar produced a usable feature window". When it stays false, pass 2, pass 3, the era counter, the checkpoint and every log line in the era-end block are ALL skipped: Train() returns having done nothing, m_eraResumePending is still false, and the next call restarts the SAME era from bar 0. That is an infinite 0->100% "scan" loop that prints absolutely nothing - the only remaining silent restart path in Train(), and it matches the reported symptom exactly. Pass 1 now counts usable vs unusable windows and reports at the pass boundary, which demonstrably executes: - total failure routes through ReportTrainStall (already capped at one line a minute, and carries the run-state flags) naming the counts, the required window width and the bar count - success prints how long the scan took and how many samples it handed to pass 2, but only once the era has passed 10s - a fast era stays as quiet as before, a slow one distinguishes "advancing" from "sweeping the same bars forever" A PARTIAL failure is normal and deliberately does not shout: pass 1 walks oldest-to-newest and the deepest bars predate the indicators' warm-up, so those windows fail and are cached as misses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:29:33 -04:00
(GetTickCount() - m_eraStartTick) / 1000.0, m_passWindowOk,
m_passWindowOk + m_passWindowFail, m_passWindowFail,
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
m_isTrainQueueCount,
CalibBandBars(totalIter, oosCutoff), CalibPurgeBars());
diag: an era that discards itself now says so instead of scanning forever add_loop is exactly "at least one bar produced a usable feature window". When it stays false, pass 2, pass 3, the era counter, the checkpoint and every log line in the era-end block are ALL skipped: Train() returns having done nothing, m_eraResumePending is still false, and the next call restarts the SAME era from bar 0. That is an infinite 0->100% "scan" loop that prints absolutely nothing - the only remaining silent restart path in Train(), and it matches the reported symptom exactly. Pass 1 now counts usable vs unusable windows and reports at the pass boundary, which demonstrably executes: - total failure routes through ReportTrainStall (already capped at one line a minute, and carries the run-state flags) naming the counts, the required window width and the bar count - success prints how long the scan took and how many samples it handed to pass 2, but only once the era has passed 10s - a fast era stays as quiet as before, a slow one distinguishes "advancing" from "sweeping the same bars forever" A PARTIAL failure is normal and deliberately does not shout: pass 1 walks oldest-to-newest and the deepest bars predate the indicators' warm-up, so those windows fail and are cached as misses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:29:33 -04:00
if(GetTickCount() - m_eraStartTick >= PASS1_LOUD_AFTER_MS)
Print(pass1Line);
else
PrintVerbose(pass1Line);
}
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
} // end if(!m_isPass2Active) - pass 1
//--- Pass 2: replay the bars pass 1 queued into m_isTrainQueue for backProp, in a freshly
//--- shuffled order - see m_isTrainQueue's declaration comment for the full rationale. Runs
//--- whenever pass 1 just finished (or we resumed straight into an already-active pass 2 - see
//--- m_isPass2Active's declaration comment); skipped on a stopped run, an era with no valid window
//--- at all (add_loop still false), or - critically - a resume into a still-unfinished pass 3 (see
//--- m_isPass2Done's declaration comment): without this last check, that resume would re-shuffle
//--- and replay the ENTIRE queue again from scratch every single call.
if(!stop && add_loop && !m_isPass2Done)
{
if(!m_isPass2Active)
{
m_isPass2Active = true;
m_isTrainCursor = 0;
feat: mini-batch gradient accumulation (F4), front-end-aware capacity budget (F6), split Wyckoff categoricals (N1) Completes the 2026-08-09 training audit. FORCES A RETRAIN of every Wyckoff-enabled config (N1 re-keys the fingerprint), and BOTH DLLs must be redeployed alongside the .ex5 - they carry new exports. F4 - mini-batch accumulation, TRAIN_BATCH_SIZE=32. Training was pure online SGD (one weight update per bar), which is the mechanical source of the era-to-era whipsaw every downstream guard was built to cope with. The O(n^2) outer product is native - AccumulateWeightGrad / AccumulateWeightGradConv / AccumulateBufferInto in Network.cl, WarriorCPU and WarriorDML - while the optimizer step is host-side MQL5 shared by all tiers (ApplyAccumToBlock), so there is one Adam/SGD implementation instead of four that can drift. - the LSTM needs no outer-product kernel (WeightsGradient already holds the sample's full dW) but could NOT simply be left un-zeroed between samples: CPU_LSTMSeqBackward/DML_LSTMSeqBackward memset it on entry. Hence a separate accumulator plus an elementwise add. - batch-norm gamma/beta accumulate in host arrays, not new BatchOptions slots - BN_OPT_STRIDE is baked into every persisted .nnw. - scoped to pass 2; online learning keeps immediate updates. Every save / checkpoint / scoring boundary flushes, scaling by the real sample count. - degrades to per-sample updates (one log line) on a tier that cannot accumulate, so old devices and DLL-free builds are unaffected. - verified offline: DirectML/batch_accum_check.cpp drives the real exports against an independent reference; at B=1 the accumulator matches the shipped unbatched kernel's own gradient to 1.1e-16. Math only - the in-situ check remains the per-layer dW/W report on a real era. F6 - ComputeFirstLayerWidth budgeted against the RAW input width even where a conv/LSTM front end had already reduced it, so an LSTM's dense stack was charged for 1,280 inputs when it receives 64. Confirmed from the deployed .cfg files: CONV, LSTM and HYBRID were all pinned at the 16-unit floor. Now budgeted against the front-end output and capped at it (never fan out), with the derivation reordered so both stages settle first. N1 - EventCode/EventPhase/StructuralPhase are signed categoricals packing direction and Wyckoff stage into one scalar across a sign discontinuity. Split into direction + [0,1] magnitude, the same convention the base OHLC block uses. Information-preserving; 13 readings now occupy 16 inputs. Compiled clean (0 errors, 0 warnings); both DLLs rebuilt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 11:48:03 -04:00
//--- MINI-BATCH ON, for pass 2 only (2026-08-09 audit, F4). Scoped this tightly on purpose:
//--- pass 2 is the only place Net.backProp() runs during era training, and everything else
//--- that ever backprops on this net - notably OnlineLearnStep, which learns from a handful of
//--- newly-confirmed live bars - wants its update applied immediately rather than held back
//--- waiting for a batch that may never fill. Switched back off where pass 2 completes.
Net.SetBatchSize(TRAIN_BATCH_SIZE);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// 2026-07-28: a "replay-only optimizer override" was removed from here. It captured every
// neuron's optimizer and forced the whole net to SGD for the duration of pass 2, on the
// rationale that oversampled minority bars should not "exploit the same Adam-style momentum
// path as the base training pass". But pass 2 IS the base training pass - it is the only place
// Net.backProp() is called during training at all (pass 1 only feeds forward and queues) - so
refactor(ai): nine class-imbalance inputs down to two The imbalance section offered nine controls for one job. Audited against the code, five of them did not do what their names said at the shipped defaults: AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns whenever the adjusted loss is on, which is default. OversampleParity DEAD in training - Training.mqh gated the replay loop on !useLogitAdjustedLoss (correctly, citing Buda et al. 2018). Live only in the online-learning path. EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma damper - "replay minority bars through pass-2 oversampling" was a focal-loss switch. ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25. UseStaticPrior An exact duplicate of FreezePriorCalibration - the two were OR'd together in the single place either is read. So they were not five mechanisms fighting; they were one mechanism plus eight knobs that mostly described machinery that no longer ran. That is worse than a real conflict, because the log agreed with the names: the label-cache line printed "reps up to 28x (90% parity) (seeding era 0's class-balance oversampling)" on every run, describing an oversampling pass that had been switched off. It is fixed here too - it cost this session a wrong diagnosis. The one genuine redundancy was focal loss, running at gamma*0.125 alongside the adjusted loss: two corrections on the same axis, the exact stacking failure this file already cited Buda et al. for in two other places, damped by a replay flag whose replay path was itself dead. Removed rather than re-tuned. The plateau ladder is unaffected - its escape is the learning-rate warm restart; the gamma anneal beside it only ever stepped toward zero. WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze: LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted- Loss boolean, since a strength dial where 0 already means off does not need an on/off switch beside it. FreezePriorCalibration unchanged. It is the only one of the six corrections with a consistency guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection already ranks on - so the loss and the deploy decision optimize one thing. The online continual-learning path keeps its own alpha-balanced focal weight, now as constants pinned to the removed inputs' shipped defaults, so its behaviour is unchanged. It legitimately needs its own correction: ApplyLogitAdjustment() only runs inside a training run, so a deployed model that was reloaded carries no logit offsets and would otherwise stream 31:1 data into itself uncorrected. The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a double fed to a %d conversion and had always emitted a literal 0; the |MR: segment is written as the constant its shipped defaults produced. Dropping either would have re-keyed every model and forced a from-scratch retrain of the one topology currently converged and trading. Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable "neutralized by prior correction" diagnostic. Both builds compile 0 errors, 0 warnings. No retrain forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
// the override applied to 100% of weight updates, not to some replay subset.
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// Adam's mt/vt were therefore never updated and its bias-correction step counter never
// advanced: TrainingOptimizer=ADAM was silently a no-op and the model trained purely on
// SGD+momentum at Adam's learning rate. It arrived with the DFA change set and was never part
// of any validated run. The optimizer the user selects is now the optimizer that runs.
// Fisher-Yates shuffle - a fresh random order every era, so Adam's momentum can't keep
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
// landing on the same contiguous same-class label run at the same point in the sequence every
// single era. Barrier labels make those runs LONGER than the old exact-pivot ones (adjacent
// bars share most of their forward window, so they usually resolve the same way), which makes
// the shuffle matter more here, not less. m_isTrainQueueWeightScale is swapped in lockstep - each
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// slot's stored per-occurrence weight (see the queueing block's oversampling comment) must
// stay attached to the same bar index it was computed for.
for(int sIdx = m_isTrainQueueCount - 1; sIdx > 0; sIdx--)
{
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//--- ShuffleRandomIndex, NOT MathRand()%: the queue routinely exceeds MathRand()'s 15-bit
//--- range on a full-history window, which silently biased this shuffle - see the helper.
int sJ = ShuffleRandomIndex(sIdx + 1);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
int sTmp = m_isTrainQueue[sIdx];
m_isTrainQueue[sIdx] = m_isTrainQueue[sJ];
m_isTrainQueue[sJ] = sTmp;
double sScaleTmp = m_isTrainQueueWeightScale[sIdx];
m_isTrainQueueWeightScale[sIdx] = m_isTrainQueueWeightScale[sJ];
m_isTrainQueueWeightScale[sJ] = sScaleTmp;
//--- the primary flag must travel with its own slot too, or the "count this bar once"
//--- marker would end up attached to a different bar's occurrence - see m_isTrainQueuePrimary
bool sPrimTmp = m_isTrainQueuePrimary[sIdx];
m_isTrainQueuePrimary[sIdx] = m_isTrainQueuePrimary[sJ];
m_isTrainQueuePrimary[sJ] = sPrimTmp;
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- the candidate id is the meta label's identity - it must stay attached to its slot
int sCandTmp = m_isTrainQueueCand[sIdx];
m_isTrainQueueCand[sIdx] = m_isTrainQueueCand[sJ];
m_isTrainQueueCand[sJ] = sCandTmp;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
}
for(; m_isTrainCursor < m_isTrainQueueCount; m_isTrainCursor++)
{
int qi = m_isTrainQueue[m_isTrainCursor];
fix: the panel showed "100%" for the whole of pass 1 The simple panel derived its percentage from pass 2's counters: (m_isTrainCursor+1) / max(m_isTrainQueueCount,1). During pass 1 those are 0 and 0, so the expression is (0+1)/max(0,1) = 100%. An era spends its first pass scanning ~38k bars - minutes of work - and the panel reported that phase as finished the entire time. Observed by the user as "started learning at 100% of their era and are stuck there", and it actively misled the diagnosis: the one number on screen said the opposite of what was happening. The UI cannot fix this on its own - it can see pass 2's counters but has no way to know which pass owns them. So each pass now PUBLISHES its own progress and a short phase name through TrainHeartbeat (which every pass already calls per item), and the panel just displays them: "learning (era 45, scan 34%)". Published before the heartbeat's 4096-item journal gate, so the panel updates continuously while the journal stays quiet. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:22:29 -04:00
TrainHeartbeat("pass 2 (shuffled backprop), sample", m_isTrainCursor + 1, m_isTrainQueueCount, "training");
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
ulong hbT = GetMicrosecondCount();
fix: the sequence models were reading the window backwards BuildFeatureWindow() replaces eight hand-rolled copies of the same loop and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first, because MQL5 timeseries indices run backwards and `r + b` with b ascending walks into the past. Harmless for PAI and CONV - a dense layer learns a weight per position either way, a conv learns time-mirrored kernels. Not harmless for the recurrent stacks: - LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t. - It writes output[] only when t == steps-1: the visible output IS the last hidden state. - c_t = f*c_{t-1} + i*g decays toward the start of the sequence. lstm_seq_flowcheck.cpp measured block 0's influence on the output at 1.2e-2 of block T-1's, at the shipped forget bias of 1.0. So the bar being PREDICTED sat at the far end of the decay and the output was handed to the OLDEST bar in the window - the exact inverse of what the window is for. ~80x backwards on LSTM and HYBRID, on all three tiers (OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never surfaced as a backend discrepancy. This does not create edge - the MI diagnostics read at the noise floor (p=0.4975) with a working positive control. It makes the one hypothesis those diagnostics explicitly do NOT cover testable: they are marginal and per-bar, and state they "cannot rule out one that only exists in combination or across time". The sequence model is the instrument for across-time structure and it has been crippled, so that hypothesis has never been honestly tested. Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and its features, so a stale .nnw would load cleanly and run a model fitted to one ordering against the other, silently. Re-keying every config is the point, not collateral damage. FORCES A FULL RETRAIN. Also: the now-relative bar caches are re-keyed on the two live paths. EnsureBarCachesCapacity() was only ever called from training paths, but once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar to RefreshConvergedSignal() and Train() is never re-entered - so nothing cleared the feature cache again for the life of the process. A chart that trained to convergence kept replaying the rows computed for the last training era's bar grid: the live signal froze at its convergence-time value, and OnlineLearnStep() backpropped those stale features against freshly resolved labels. Backtests were never affected (an inference-only process never allocates the arrays, so every read recomputes). Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 18:28:44 -04:00
bool qWindowOk = BuildFeatureWindow(qi);
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- Meta target: the input is window + per-candidate setup descriptor; the net's input layer
//--- is sized for both (NetInputWidth), so the append must happen before EVERY forward.
if(qWindowOk && IsMetaTarget())
AppendCandidateFeatures(m_isTrainQueueCand[m_isTrainCursor]);
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
m_passFeatUs += GetMicrosecondCount() - hbT;
fix: improve error messages and suppress false sharing-violation logs - BufferDouble: replace hardcoded "DirectML/CPU-DLL" with dynamic backend name and add buffer index/element count to all error prints for easier debugging. - NetPersistence: distinguish missing file from transient lock by probing FileIsExist before logging, eliminating false "sharing violation" warnings when no saved model exists on first run.
2026-08-02 01:09:18 -04:00
//--- A failed forward pass must NOT be followed by backProp() further down this block: the
//--- output layer would still hold the PREVIOUS sample's activations, so the update would be
//--- this bar's label against another bar's prediction - training on pure noise while every
//--- accuracy counter kept reporting normally.
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
hbT = GetMicrosecondCount();
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
bool qForwardOk = (qWindowOk && TempData.Total() >= NetInputWidth() &&
fix: improve error messages and suppress false sharing-violation logs - BufferDouble: replace hardcoded "DirectML/CPU-DLL" with dynamic backend name and add buffer index/element count to all error prints for easier debugging. - NetPersistence: distinguish missing file from transient lock by probing FileIsExist before logging, eliminating false "sharing violation" warnings when no saved model exists on first run.
2026-08-02 01:09:18 -04:00
Net.feedForward(TempData));
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
m_passNetUs += GetMicrosecondCount() - hbT;
fix: improve error messages and suppress false sharing-violation logs - BufferDouble: replace hardcoded "DirectML/CPU-DLL" with dynamic backend name and add buffer index/element count to all error prints for easier debugging. - NetPersistence: distinguish missing file from transient lock by probing FileIsExist before logging, eliminating false "sharing violation" warnings when no saved model exists on first run.
2026-08-02 01:09:18 -04:00
if(qWindowOk && !qForwardOk && !forwardFailureReported)
{
forwardFailureReported = true;
Print(__FUNCTION__ + ": CNet::feedForward FAILED at era " + IntegerToString((int)m_eraCount) +
" - this era's remaining samples are being skipped, not trained. A layer is refusing to"
" accept its own output (check the preceding BufferWrite/BufferRead lines for which"
" buffer, and see NormalizeHost in AI\\NeuronBatchNorm.mqh for the batch-norm case).");
}
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- META TARGET pass 2: binary win/loss backprop per candidate. No excursion step (that head
//--- belongs to the direction models), no arrows, no 3-class softmax - just the running IS
//--- stats under the win->Buy / loss->Sell mapping documented at pass 1's meta branch.
if(qForwardOk && IsMetaTarget())
{
int qc = m_isTrainQueueCand[m_isTrainCursor];
Net.getResults(TempData);
double qPwin = MetaWinProbability();
bool qWon = MetaCandidateWon(qc, qi);
//--- the argmax of a 2-class softmax IS pWin >= 0.5 - the unthresholded "call"
bool qCall = (qPwin >= 0.5);
bool qHit = (qCall == qWon);
if(qHit)
dForecast += (100 - dForecast) / Net.recentAverageSmoothingFactor;
else
dForecast -= dForecast / Net.recentAverageSmoothingFactor;
dUndefine -= dUndefine / Net.recentAverageSmoothingFactor;
if(qCall)
m_countBuySignals++;
else
m_countSellSignals++;
//--- persistent IS precision over the candidates the model would trade, in WINS - the
//--- meta analogue of the direction path's m_cumIsTotal contract (compared against the OOS
//--- side as the overfitting signal, so both must count the same quantity).
if(qCall)
{
m_cumIsTotal++;
if(qWon)
m_cumIsCorrect++;
}
UpdateTrainingStatusLabel(
StringFormat("Training candidate %d of %d -> %.2f%% (shuffled)", m_isTrainCursor + 1,
m_isTrainQueueCount,
(double)(m_isTrainCursor + 1.0) / MathMax(m_isTrainQueueCount, 1) * 100),
(TempData.Total() > 0) ? TempData[0] : 0.0,
(TempData.Total() > 1) ? TempData[1] : 0.0, 0.0, qPwin);
TempData.Clear();
//--- slot 0 = P(win), slot 1 = P(loss); same label smoothing as the 3-class head
TempData.Add(qWon ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW);
TempData.Add(qWon ? LABEL_SMOOTH_LOW : LABEL_SMOOTH_HIGH);
ulong hbBpM = GetMicrosecondCount();
Net.backProp(TempData, 1.0);
m_passNetUs += GetMicrosecondCount() - hbBpM;
}
else
if(qForwardOk)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
feat: excursion-size head (Stage 1, measurement only) Direction is closed - normalised asymmetry fails on three instruments with a working positive control, and the classifier's own best-of-999 era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is a different question and RANGE clears at ~4x its null. Checked the denomination before building on that, since the source memo warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is predictable" is a claim about travel RELATIVE to current ATR, not a restatement of "ATR is autocorrelated". It is exactly the part a fixed multiple (stop 3.31*ATR, target 1.64*ATR) discards. A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches ladder rung k) upward and downward. Survival parameterisation rather than regressing the multiple, because it needs nothing new from CNet: sigmoid outputs and the per-neuron delta the `total != 3` branch already applies (a quantile head would need a linear activation and a pinball gradient in Network.mqh, Network.cl and the DirectML path, on a class four topologies share). Targets are free - m_ladderUpAt already records first-touch age per rung with 0 meaning never reached. Separate net, not extra outputs on the classifier: more outputs would change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push the count off 3 - the exact condition backProp uses to select the joint softmax gradient the 3-class head depends on. The classifier is bit-for-bit unaffected and this is removable without trace. STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the constant per-rung base rate - the baseline a fixed ATR multiple already assumes - with both predictors fitted IS and evaluated OOS, so neither gets a look at the test set. Positive skill justifies Stage 2 (drive SL/TP and sizing off ExcursionQuantile, which is defined and deliberately uncalled). Zero or negative means ATR already carries everything and Stage 2 must not be built. Trains only on primary occurrences: the replay queue oversamples for CLASS balance, and a direction-balanced sample is a biased SIZE sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
//--- EXCURSION HEAD, trained here and ONLY here in pass 2. Must run BEFORE getResults(),
//--- which overwrites TempData in place with the classifier's output activations - the
//--- feature window is gone after the next line. Only primary occurrences: the replay queue
//--- oversamples for CLASS balance, and duplicating minority-direction bars would skew the
//--- excursion-size distribution the head is trying to learn (same correction m_cumIsTotal
//--- makes, for a target where it matters even more - size and direction are unrelated, so
//--- a direction-balanced sample is a biased size sample).
if(m_isTrainQueuePrimary[m_isTrainCursor])
ExcursionTrainStep(qi);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
Net.getResults(TempData);
// Must go through ApplyClassificationSoftmax() (3-output case) before reading the
// per-class values below - Net.getResults() returns each output neuron's own independent
// SIGMOID activation (each already in [0,1] but NOT summing to 1 across the three), not a
// true class-conditional probability distribution; ApplyClassificationSoftmax() is what
// turns that into one (and is also what pass 1/3's displayNeuron0/1/2 already go through).
double qPrevSignal = (m_outputNeuronsCount == 3) ? ApplyClassificationSoftmax() : TempData[0];
double pt0 = (TempData.Total() > 0) ? TempData[0] : 0.0;
double pt1 = (TempData.Total() > 1) ? TempData[1] : 0.0;
double pt2 = (TempData.Total() > 2) ? TempData[2] : 0.0;
bool qBuy = m_labelCacheHasValue[qi] ? m_labelCacheBuy[qi] : false;
bool qSell = m_labelCacheHasValue[qi] ? m_labelCacheSell[qi] : false;
ENUM_SIGNAL qTrueSignal = qBuy ? Buy : (qSell ? Sell : Neutral);
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
//--- Per-direction outcomes for this bar - see m_oosBuyPredictedWins. Needed on the IS side
//--- too: the operating point is FITTED here and GRADED by the OOS gate, so if the two
//--- optimise different quantities the threshold is tuned for the wrong objective.
bool qWinLong = (m_labelCacheHasValue[qi] && qi < ArraySize(m_winLongCache))
? m_winLongCache[qi] : false;
bool qWinShort = (m_labelCacheHasValue[qi] && qi < ArraySize(m_winShortCache))
? m_winShortCache[qi] : false;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
UpdateTrainingStatusLabel(
StringFormat("Training bar %d of %d -> %.2f%% (shuffled)", m_isTrainCursor + 1, m_isTrainQueueCount, (double)(m_isTrainCursor + 1.0) / MathMax(m_isTrainQueueCount, 1) * 100),
pt0, pt1, pt2, qPrevSignal);
//--- Predicted-signal tally, chart marker, and IS-accuracy stat that pass 1 used to compute
//--- from its own (now-removed) redundant feedForward on this same bar - see pass 1's
//--- wouldQueue comment. Uses THIS feedForward's result (the only one this bar gets), so
//--- these now reflect the model's state as of this bar's own turn in the shuffled replay
//--- (post any earlier-shuffled bar's backProp this era), not a separate pre-training
//--- snapshot - matching how a standard shuffled-epoch SGD run reports running training
//--- accuracy during the epoch rather than in a discarded pre-epoch dry run.
switch(DoubleToSignal(qPrevSignal))
{
case Buy:
m_countBuySignals++;
break;
case Sell:
m_countSellSignals++;
break;
default:
m_countNeutralSignals++;
break;
}
datetime qBarTime = m_Time.GetData(qi);
// NMS on: record only (the era-end sweep renders); off: draw inline. See pass 1's note.
if(m_signalClusterWindow > 0)
{
if(qi < ArraySize(m_arrowSignalCache))
m_arrowSignalCache[qi] = qPrevSignal;
}
else
if(DoubleToSignal(qPrevSignal) == Neutral)
DeleteObject(qBarTime);
else
feat(chart): signal marks become price LEVELS at the trigger, not arrows beside the candle User request: 'move from arrows on lows and highs to small horizontal lines at the actual prices the entry/exit would trigger, just a bit larger than the candles. dark green for buy, dark red for sell.' Every mark is now an OBJ_TREND segment with both anchors at one price and both rays off, spanning 1.3 bar widths, drawn at the bar's CLOSE - the price a market order actually fires at, and the exact entry TripleBarrierLabel assumes. It used to sit on the candle's LOW for a Buy and its HIGH for a Sell: prices the trade never touches, picked so an arrow glyph would clear the candle. The tooltip now carries that price too. COLOUR NOW MEANS DIRECTION AND ONLY DIRECTION on every layer (dark green / dark red). Layer moves to width+style - the traded vote is solid and thick and drawn in front, a single model's raw opinion is thin, dotted and behind the candles - which keeps the distinction the old palette existed to draw (a model's opinion must never read as a trade) while freeing colour to say one thing consistently. Consequences handled, all of them the same 'a typed scan went blind' failure: - SaveChartSignals filtered OBJPROP_TYPE == OBJ_ARROW and read OBJPROP_ARROWCODE. It now filters OBJ_TREND and recovers direction from the colour. The sidecar keeps the old 217/218 numbers as its buy/sell token deliberately, so existing .arrows files still load. - AdvanceChartSignalRestore now rebuilds through the SAME creation point the live path uses, so a restored mark and a fresh one are identical objects. - The rescan-scoped delete enumerated ObjectsTotal(OBJ_ARROW) - retyped, or it silently deletes nothing. - ApplySignalsVisibility enumerated OBJ_ARROW with NO prefix filter. Under the new type that would have hidden and shown THE USER'S OWN trend lines on every Hide/Show click; it is now prefix-scoped. The old type was uncommon enough on a real chart to mask the missing check - trend lines are the most hand-drawn object there is. - DrawObject's high/low parameters are gone (6 call sites pass m_Close instead), so no caller can hand it a price it no longer draws at. - Fixed a pre-existing stale comment that still described the purge sweep as OBJ_ARROW-only three lines above the note explaining it had been widened to every type. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 14:11:49 -04:00
DrawObject(qBarTime, qPrevSignal, m_Close.GetData(qi));
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
bool qClassified = (DoubleToSignal(qPrevSignal) == Buy || DoubleToSignal(qPrevSignal) == Sell || DoubleToSignal(qPrevSignal) == Neutral);
if(qClassified)
{
bool isHit = (DoubleToSignal(qPrevSignal) == qTrueSignal);
if(isHit)
dForecast += (100 - dForecast) / Net.recentAverageSmoothingFactor;
else
dForecast -= dForecast / Net.recentAverageSmoothingFactor;
dUndefine -= dUndefine / Net.recentAverageSmoothingFactor;
//--- Compounded, persistent DIRECTIONAL win-rate: count only bars the model actually called
//--- Buy or Sell (a Neutral "no trade" call is neither a win nor a loss), so this tracks the
//--- accuracy of its directional signals rather than the Neutral-inflated all-class rate.
//--- ...and count each BAR once, not each oversampled OCCURRENCE (m_isTrainQueuePrimary):
//--- the queue duplicates minority bars up to ~21x, so counting every occurrence scored this
//--- metric over a ~58%-directional set while its OOS counterpart scored the real ~6%
//--- distribution - two numbers that look comparable, aren't, and made a healthy run read as
//--- severe overfitting. See m_isTrainQueuePrimary for the worked example.
ENUM_SIGNAL qPred = DoubleToSignal(qPrevSignal);
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
//--- Did the implied trade pay? Same distinction as the OOS side - see
//--- m_oosBuyPredictedWins - and it has to be made identically on both, because the IS and
//--- OOS win rates are read side by side as the overfitting signal. Measuring one in wins
//--- and the other in label agreement would put a fixed gap between them that has nothing
//--- to do with generalization.
bool qTradeWon = (qPred == Buy) ? qWinLong : ((qPred == Sell) ? qWinShort : false);
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
if(m_isTrainQueuePrimary[m_isTrainCursor] && (qPred == Buy || qPred == Sell))
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
m_cumIsTotal++;
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
if(qTradeWon)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_cumIsCorrect++;
}
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- THE OPERATING-POINT FIT NO LONGER HARVESTS HERE. It used to, on the argument that
//--- pass 2's forward pass made the margin free - which was true, and irrelevant: these
//--- are the bars the very next line backprops on, so within a handful of eras the
//--- histogram describes memorized behaviour and not the model's behaviour on unseen
//--- bars. It moved to the held-out calibration walk below; DIR_CONF_CALIB_PCT_OF_IS
//--- carries the measured IS-vs-OOS divergence that forced the move.
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
else
if(qBuy && qSell)
dUndefine += (100 - dUndefine) / Net.recentAverageSmoothingFactor;
TempData.Clear();
if(m_outputNeuronsCount == 1)
TempData.Add(qBuy && !qSell ? 1 : !qBuy && qSell ? -1 : 0);
else
if(m_outputNeuronsCount == 3)
{
TempData.Add(qBuy ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW);
TempData.Add(qSell ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW);
TempData.Add((!qBuy && !qSell) ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW);
}
refactor(ai): nine class-imbalance inputs down to two The imbalance section offered nine controls for one job. Audited against the code, five of them did not do what their names said at the shipped defaults: AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns whenever the adjusted loss is on, which is default. OversampleParity DEAD in training - Training.mqh gated the replay loop on !useLogitAdjustedLoss (correctly, citing Buda et al. 2018). Live only in the online-learning path. EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma damper - "replay minority bars through pass-2 oversampling" was a focal-loss switch. ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25. UseStaticPrior An exact duplicate of FreezePriorCalibration - the two were OR'd together in the single place either is read. So they were not five mechanisms fighting; they were one mechanism plus eight knobs that mostly described machinery that no longer ran. That is worse than a real conflict, because the log agreed with the names: the label-cache line printed "reps up to 28x (90% parity) (seeding era 0's class-balance oversampling)" on every run, describing an oversampling pass that had been switched off. It is fixed here too - it cost this session a wrong diagnosis. The one genuine redundancy was focal loss, running at gamma*0.125 alongside the adjusted loss: two corrections on the same axis, the exact stacking failure this file already cited Buda et al. for in two other places, damped by a replay flag whose replay path was itself dead. Removed rather than re-tuned. The plateau ladder is unaffected - its escape is the learning-rate warm restart; the gamma anneal beside it only ever stepped toward zero. WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze: LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted- Loss boolean, since a strength dial where 0 already means off does not need an on/off switch beside it. FreezePriorCalibration unchanged. It is the only one of the six corrections with a consistency guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection already ranks on - so the loss and the deploy decision optimize one thing. The online continual-learning path keeps its own alpha-balanced focal weight, now as constants pinned to the removed inputs' shipped defaults, so its behaviour is unchanged. It legitimately needs its own correction: ApplyLogitAdjustment() only runs inside a training run, so a deployed model that was reloaded carries no logit offsets and would otherwise stream 31:1 data into itself uncorrected. The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a double fed to a %d conversion and had always emitted a literal 0; the |MR: segment is written as the constant its shipped defaults produced. Dropping either would have re-keyed every model and forced a from-scratch retrain of the one topology currently converged and trading. Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable "neutralized by prior correction" diagnostic. Both builds compile 0 errors, 0 warnings. No retrain forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
// Per-slot weight from m_isTrainQueueWeightScale[m_isTrainCursor], decided once at queue
// time in pass 1. Currently always 1.0: class imbalance is corrected analytically inside
// the gradient by the logit-adjusted loss, so there is no per-sample reweighting left to
// apply here at all. Kept as a real per-slot value rather than a literal 1.0 inline so a
// future supplemental weight can be reintroduced without re-touching the queueing or
// shuffle code.
//--- FOCAL-LOSS MODULATION REMOVED 2026-07-31. It multiplied this weight by (1-pt)^gamma,
//--- a second correction on the same axis as the logit adjustment - the stacking failure
//--- Buda et al. 2018 describes and this file already cited in two other places. It was
//--- running at an eighth strength (gamma * 0.125), damped by the replay toggle, for a
//--- replay path that the adjusted loss had already switched off - so the damping was
//--- calibrated against a mechanism that was not running. See the class-imbalance audit in
//--- Variables\Inputs.mqh.
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
double qSampleWeight = m_isTrainQueueWeightScale[m_isTrainCursor];
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
ulong hbBp = GetMicrosecondCount();
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
Net.backProp(TempData, qSampleWeight);
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
m_passNetUs += GetMicrosecondCount() - hbBp;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
fix(deinit): vote arrows survived the cheap sweep, and 5 long loops ignored the stop Leftover chart objects on long-history charts. Two causes, one of them introduced by 07aa017. THE ONE I ADDED. The filtered view's overlay draws up to SIGNAL_RESCAN_LOOKBACK_BARS vote arrows. OnDeinit's EARLY VISIBLE-UI SWEEP runs with skipArrows=true, which skips any prefix equal to SIG_ARROW_PREFIX - and "WarSig_VOTE_..." starts with "WarSig_", so every one of them was skipped by the one sweep that is cheap enough to always complete. They then sat in the object list while the two expensive scans that follow walked it: a per-member SaveChartSignals O(total) scan, then the by-name rescan. On a chart with years of history that is thousands of extra objects walked twice, inside a teardown budget measured from the stop REQUEST rather than from OnDeinit's first line. skipArrows exists because the per-model arrows' sidecar is rebuilt by SCANNING them off the chart, so they cannot be deleted before that write. Vote arrows have no sidecar - they are a reconstruction, rebuilt on the next attach - so nothing is preserving them and they now get their own prefix slot, deleted by one native call in the first few milliseconds. THE FIVE LOOPS. A time budget bounds THROUGHPUT, not latency to an unload, and OnDeinit cannot begin until whatever is in flight returns. These all scaled with history and none of them checked: * Training passes 2, 2.5 and 3 yielded only on TRAIN_TIME_BUDGET_MS. Pass 1 has checked IsStopped() all along; the other three never have, and they are the ones that grow with the bar count. Free to fix - the resume state is written either way, so a stopped chunk simply is not re-entered. * PruneDirectionalClusters: the one UNCHUNKED sweep left, once per era over every bar, with its own header noting that raising the training budget cannot help its cost. Now bails outright. * AdvanceChartSignalRestore / AdvanceChartSignalRescan: chunked, but the rescan runs a full feedForward per bar over up to 5000 bars and the restore can hold MAX_RESTORED_ARROWS entries. Checked on the same 64-object stride as the clock read, since the check is not free either. * AdvanceFilteredOverlay (mine, 07aa017) replays Direction() on every classic filter per bar and had no check at all. Now per bar. ChartUI.mqh had ZERO shutdown checks across six loops before this. Nothing was added to the purge path itself: that is the work that must complete, and an IsStopped() check inside it would abort unconditionally - IsStopped() is already true by the time OnDeinit runs. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 17:13:50 -04:00
//--- YIELD ON TIME **OR** ON A STOP REQUEST. The time budget bounds THROUGHPUT; it does not
//--- bound LATENCY to an unload. MetaTrader measures its ~4,500 ms teardown budget from the
//--- stop request and OnDeinit cannot start until whatever is in flight returns, so a chunk
//--- that keeps training for its full slice after _StopFlag is raised spends that time out of
//--- the chart cleanup - which is what strands arrows and panels (see OnDeinit's ordering
//--- notes). Pass 1 has checked IsStopped() all along; passes 2, 2.5 and 3 never did, and
//--- they are the ones that grow with history. Yielding here is free: the resume state below
//--- is written either way, so a stopped chunk simply never gets re-entered.
if(m_isTrainCursor + 1 < m_isTrainQueueCount && (IsStopped() || GetTickCount() - chunkStartTick >= TRAIN_TIME_BUDGET_MS))
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
//--- yield: save enough to resume PASS 2 mid-queue on the next call - m_isPass2Active
//--- and m_isTrainCursor (both members) carry the actual resume position; bars/oosCutoff/
//--- add_loop are stashed the same way pass 1 already does, since era-end logic just
//--- below still needs them once pass 2 finishes.
m_resumeBars = bars;
m_resumeTotalIter = totalIter;
m_resumeOosCutoff = oosCutoff;
m_resumeAddLoop = add_loop;
m_resumeBarIndex = i;
m_eraResumePending = true;
m_modelEta = eta;
return;
}
}
feat: mini-batch gradient accumulation (F4), front-end-aware capacity budget (F6), split Wyckoff categoricals (N1) Completes the 2026-08-09 training audit. FORCES A RETRAIN of every Wyckoff-enabled config (N1 re-keys the fingerprint), and BOTH DLLs must be redeployed alongside the .ex5 - they carry new exports. F4 - mini-batch accumulation, TRAIN_BATCH_SIZE=32. Training was pure online SGD (one weight update per bar), which is the mechanical source of the era-to-era whipsaw every downstream guard was built to cope with. The O(n^2) outer product is native - AccumulateWeightGrad / AccumulateWeightGradConv / AccumulateBufferInto in Network.cl, WarriorCPU and WarriorDML - while the optimizer step is host-side MQL5 shared by all tiers (ApplyAccumToBlock), so there is one Adam/SGD implementation instead of four that can drift. - the LSTM needs no outer-product kernel (WeightsGradient already holds the sample's full dW) but could NOT simply be left un-zeroed between samples: CPU_LSTMSeqBackward/DML_LSTMSeqBackward memset it on entry. Hence a separate accumulator plus an elementwise add. - batch-norm gamma/beta accumulate in host arrays, not new BatchOptions slots - BN_OPT_STRIDE is baked into every persisted .nnw. - scoped to pass 2; online learning keeps immediate updates. Every save / checkpoint / scoring boundary flushes, scaling by the real sample count. - degrades to per-sample updates (one log line) on a tier that cannot accumulate, so old devices and DLL-free builds are unaffected. - verified offline: DirectML/batch_accum_check.cpp drives the real exports against an independent reference; at B=1 the accumulator matches the shipped unbatched kernel's own gradient to 1.1e-16. Math only - the in-situ check remains the per-layer dW/W report on a real era. F6 - ComputeFirstLayerWidth budgeted against the RAW input width even where a conv/LSTM front end had already reduced it, so an LSTM's dense stack was charged for 1,280 inputs when it receives 64. Confirmed from the deployed .cfg files: CONV, LSTM and HYBRID were all pinned at the 16-unit floor. Now budgeted against the front-end output and capped at it (never fan out), with the derivation reordered so both stages settle first. N1 - EventCode/EventPhase/StructuralPhase are signed categoricals packing direction and Wyckoff stage into one scalar across a sign discontinuity. Split into direction + [0,1] magnitude, the same convention the base OHLC block uses. Information-preserving; 13 readings now occupy 16 inputs. Compiled clean (0 errors, 0 warnings); both DLLs rebuilt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 11:48:03 -04:00
//--- Apply whatever the final (usually short) batch of this era accumulated, and return the net
//--- to per-sample updates. MUST happen before pass 3 scores anything: the selection metric has
//--- to describe weights with no unapplied gradients sitting behind them, and the checkpoint
//--- taken from that score has to be the same model. FlushBatch scales by the REAL sample count,
//--- so a short trailing batch still takes a correctly-sized step.
Net.FlushBatch();
Net.SetBatchSize(1);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_isPass2Active = false;
m_isPass2Done = true;
}
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- Pass 2.5: the CALIBRATION walk. Scores the held-out band (see CalibLoIndex for the layout) with
//--- the weights pass 2 just finished training, harvests the margin histogram, and fits this era's
//--- operating point - all BEFORE pass 3 grades anything.
//---
//--- Three properties have to hold at once and only this position gives all three:
//--- not trained on - pass 1 kept the band out of the backprop queue, so the histogram measures
//--- generalization rather than memorization (the failure that moved it here)
//--- not graded - pass 3's OOS window is disjoint from the band, so the numbers the deploy
//--- gate ranks are still produced by a threshold that never saw them
//--- current weights - after pass 2, so the operating point belongs to the weights it will be
//--- applied to; the margin distribution moves with them every era
//---
//--- Batch norm is frozen for the walk exactly as pass 3 freezes it, and for the same reason: an
//--- unfrozen BN would let the running statistics drift while scoring, so the fitted threshold would
//--- describe a slightly different function than the one pass 3 then grades.
if(!stop && add_loop && !m_isCalibDone)
{
int calibLo = CalibLoIndex(oosCutoff);
int calibHi = CalibHiIndex(totalIter, oosCutoff);
if(!m_isCalibActive)
{
m_isCalibActive = true;
Net.SetBatchNormFrozen(true);
ResetDirConfHistogram();
//--- Same upper clamp pass 3 applies: a bar needs m_historyBars of older bars behind it to
//--- build a window at all, so the band is trimmed to what is actually scoreable.
m_calibStartIndex = (int)MathMin(calibHi - 1, bars - MathMax(m_historyBars, 0) - 2);
m_calibIndex = m_calibStartIndex;
}
for(; m_calibIndex >= calibLo; m_calibIndex--)
{
int ci = m_calibIndex;
//--- Same eligibility test pass 1 gates labelling on (its line reads
//--- `i < bars-historyBars-1 && i > 1 && Time[i] > dtStudied`), so this walk can only score bars
//--- pass 1 actually produced a label for. Pass 3 applies the identical test on its own window.
if(!(ci < (int)(bars - MathMax(m_historyBars, 0) - 1) && ci > 1 && m_Time.GetData(ci) > dtStudied))
continue;
TrainHeartbeat("pass 2.5 (calibration), bar", m_calibStartIndex - m_calibIndex + 1,
m_calibStartIndex - calibLo + 1, "calibrating");
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- META TARGET: harvest one histogram sample per CANDIDATE in the band - margin is P(win),
//--- outcome is the candidate's own triple-barrier win. The window must be rebuilt per
//--- candidate because the appended descriptor differs; the bar features behind it come from
//--- the feature cache, so the rebuild is cheap. Bars without candidates contribute nothing -
//--- the coverage denominator (m_dirConfPrimaryBars) is CANDIDATES, matching the coverage
//--- numerator the threshold admits, and FitDirConfThreshold's coverage x (precision -
//--- break-even) objective is exactly the design doc's operating point for the meta head.
if(IsMetaTarget())
{
for(int cd = MetaCandFirst(ci); cd >= 0; cd = MetaCandNext(cd))
{
ulong hbM = GetMicrosecondCount();
bool mWindowOk = BuildFeatureWindow(ci);
if(mWindowOk)
AppendCandidateFeatures(cd);
m_passFeatUs += GetMicrosecondCount() - hbM;
hbM = GetMicrosecondCount();
bool mForwardOk = (mWindowOk && TempData.Total() >= NetInputWidth() &&
Net.feedForward(TempData));
m_passNetUs += GetMicrosecondCount() - hbM;
if(!mForwardOk)
break;
Net.getResults(TempData);
AccumulateDirConfSample(MetaWinProbability(), MetaCandidateWon(cd, ci), true);
}
}
else
{
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
ulong hbC = GetMicrosecondCount();
bool cWindowOk = BuildFeatureWindow(ci);
m_passFeatUs += GetMicrosecondCount() - hbC;
hbC = GetMicrosecondCount();
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
bool cForwardOk = (cWindowOk && TempData.Total() >= NetInputWidth() &&
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
Net.feedForward(TempData));
m_passNetUs += GetMicrosecondCount() - hbC;
if(cForwardOk)
{
Net.getResults(TempData);
//--- RAW argmax softmax, NOT AdjustedSignalFromSoftmax(): feeding the fit its own already-
//--- thresholded decisions would make the threshold a fixed point of itself, able only to
//--- ratchet upward. ApplyClassificationSoftmax() leaves the probabilities in TempData,
//--- which is what DirectionalMargin() reads.
double cSignal = (m_outputNeuronsCount == 3) ? ApplyClassificationSoftmax() : TempData[0];
ENUM_SIGNAL cPred = DoubleToSignal(cSignal);
//--- Scored in WINS - did the trade this call implies actually pay - not in agreement with
//--- the collapsed 3-class label. Same distinction pass 3 makes (see m_oosBuyPredictedWins);
//--- the two must be measured identically or the operating point is chosen for one quantity
//--- and graded on another.
bool cWinLong = (m_labelCacheHasValue[ci] && ci < ArraySize(m_winLongCache))
? m_winLongCache[ci] : false;
bool cWinShort = (m_labelCacheHasValue[ci] && ci < ArraySize(m_winShortCache))
? m_winShortCache[ci] : false;
bool cTradeWon = (cPred == Buy) ? cWinLong : ((cPred == Sell) ? cWinShort : false);
//--- isPrimaryBar is unconditionally true: this walk visits each bar once in chronological
//--- order, so there is no oversampled replay to correct for here.
AccumulateDirConfSample(DirectionalMargin(), cTradeWon, true);
perf: pass 1 forward-passed ~40% of bars that a later pass redid anyway Pass 1 already skipped its feedForward on QUEUED bars, because pass 2 redoes them. The same argument covers two more bands it was still forwarding: OOS window (30% of bars) - pass 3 re-forwards every one of them calibration band (~10% of bars) - pass 2.5 re-forwards every one of them All three passes derive their bounds from the same helpers and apply the identical eligibility test, so the bar sets are equal by construction, not by coincidence. Only the two purge bands and the ineligible edge bars are visited in pass 1 and nowhere else - those keep their forward pass. The scan's copy was never the one that survived. Its arrow-cache write was overwritten by pass 3's (with the thresholded, post-training decision), its status-label paint was transient, and its predicted-class tally measured last era's weights. Those tallies move to pass 2.5 and pass 3, on the raw argmax exactly as pass 1 and pass 2 count it, so the population behind the panel's "Predicted -> Buy/Sell/Neutral" line is unchanged and stays comparable with the "Actual" line beside it, which pass 1 still accumulates over every labelled bar. Verified unaffected by the cut: dPrevSignal and m_lastBarTime are both written last by bars 0/1, which are label-ineligible and therefore still forwarded, so FinalizeTrainRun's `dtStudied = m_lastBarTime` and Lifecycle's newBarPending sentinel read the same values as before. Correctness, not just speed: batch norm is UNFROZEN during pass 1 (passes 2.5 and 3 freeze it deliberately), so every scan-time forward on a held-out bar was advancing the BN running mean/variance from data the model is graded on. Those running statistics are inference-time model state. It is the mild, unsupervised kind of leakage - feature statistics, not labels - but it fed the weights pass 3 then scored, and it is now gone. Cost: ~40% of all bars lose one forward pass per era, ~16% of net time once pass 2's backward pass is weighted in. Per-dispatch, so it lands on every backend. Both variants compile 0 errors / 0 warnings. Build tag scan-nofwd-v5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:50:14 -04:00
//--- Predicted-class tally and chart marker for the calibration band, which pass 1 used to
//--- compute from its own (now-removed) redundant feedForward on this same bar. The panel
//--- reads these counts directly against m_trueBuyCount/m_trueSellCount/m_trueNeutralCount,
//--- which pass 1 still accumulates over EVERY labelled bar - so the predicted side has to
//--- keep spanning the same bars or the two lines stop being comparable. Same value pass 1
//--- used (raw argmax, not the thresholded decision), so only the weights differ: these are
//--- post-training now, matching what pass 2 already does for the queued bars.
switch(cPred)
{
case Buy:
m_countBuySignals++;
break;
case Sell:
m_countSellSignals++;
break;
default:
m_countNeutralSignals++;
break;
}
datetime cBarTime = m_Time.GetData(ci);
if(ci > 0)
{
// NMS on: record only (the era-end sweep renders); off: draw inline. See pass 1's note.
if(m_signalClusterWindow > 0)
{
if(ci < ArraySize(m_arrowSignalCache))
m_arrowSignalCache[ci] = cSignal;
}
else
if(cPred == Neutral)
DeleteObject(cBarTime);
else
feat(chart): signal marks become price LEVELS at the trigger, not arrows beside the candle User request: 'move from arrows on lows and highs to small horizontal lines at the actual prices the entry/exit would trigger, just a bit larger than the candles. dark green for buy, dark red for sell.' Every mark is now an OBJ_TREND segment with both anchors at one price and both rays off, spanning 1.3 bar widths, drawn at the bar's CLOSE - the price a market order actually fires at, and the exact entry TripleBarrierLabel assumes. It used to sit on the candle's LOW for a Buy and its HIGH for a Sell: prices the trade never touches, picked so an arrow glyph would clear the candle. The tooltip now carries that price too. COLOUR NOW MEANS DIRECTION AND ONLY DIRECTION on every layer (dark green / dark red). Layer moves to width+style - the traded vote is solid and thick and drawn in front, a single model's raw opinion is thin, dotted and behind the candles - which keeps the distinction the old palette existed to draw (a model's opinion must never read as a trade) while freeing colour to say one thing consistently. Consequences handled, all of them the same 'a typed scan went blind' failure: - SaveChartSignals filtered OBJPROP_TYPE == OBJ_ARROW and read OBJPROP_ARROWCODE. It now filters OBJ_TREND and recovers direction from the colour. The sidecar keeps the old 217/218 numbers as its buy/sell token deliberately, so existing .arrows files still load. - AdvanceChartSignalRestore now rebuilds through the SAME creation point the live path uses, so a restored mark and a fresh one are identical objects. - The rescan-scoped delete enumerated ObjectsTotal(OBJ_ARROW) - retyped, or it silently deletes nothing. - ApplySignalsVisibility enumerated OBJ_ARROW with NO prefix filter. Under the new type that would have hidden and shown THE USER'S OWN trend lines on every Hide/Show click; it is now prefix-scoped. The old type was uncommon enough on a real chart to mask the missing check - trend lines are the most hand-drawn object there is. - DrawObject's high/low parameters are gone (6 call sites pass m_Close instead), so no caller can hand it a price it no longer draws at. - Fixed a pre-existing stale comment that still described the purge sweep as OBJ_ARROW-only three lines above the note explaining it had been widened to every type. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 14:11:49 -04:00
DrawObject(cBarTime, cSignal, m_Close.GetData(ci));
perf: pass 1 forward-passed ~40% of bars that a later pass redid anyway Pass 1 already skipped its feedForward on QUEUED bars, because pass 2 redoes them. The same argument covers two more bands it was still forwarding: OOS window (30% of bars) - pass 3 re-forwards every one of them calibration band (~10% of bars) - pass 2.5 re-forwards every one of them All three passes derive their bounds from the same helpers and apply the identical eligibility test, so the bar sets are equal by construction, not by coincidence. Only the two purge bands and the ineligible edge bars are visited in pass 1 and nowhere else - those keep their forward pass. The scan's copy was never the one that survived. Its arrow-cache write was overwritten by pass 3's (with the thresholded, post-training decision), its status-label paint was transient, and its predicted-class tally measured last era's weights. Those tallies move to pass 2.5 and pass 3, on the raw argmax exactly as pass 1 and pass 2 count it, so the population behind the panel's "Predicted -> Buy/Sell/Neutral" line is unchanged and stays comparable with the "Actual" line beside it, which pass 1 still accumulates over every labelled bar. Verified unaffected by the cut: dPrevSignal and m_lastBarTime are both written last by bars 0/1, which are label-ineligible and therefore still forwarded, so FinalizeTrainRun's `dtStudied = m_lastBarTime` and Lifecycle's newBarPending sentinel read the same values as before. Correctness, not just speed: batch norm is UNFROZEN during pass 1 (passes 2.5 and 3 freeze it deliberately), so every scan-time forward on a held-out bar was advancing the BN running mean/variance from data the model is graded on. Those running statistics are inference-time model state. It is the mild, unsupervised kind of leakage - feature statistics, not labels - but it fed the weights pass 3 then scored, and it is now gone. Cost: ~40% of all bars lose one forward pass per era, ~16% of net time once pass 2's backward pass is weighted in. Per-dispatch, so it lands on every backend. Both variants compile 0 errors / 0 warnings. Build tag scan-nofwd-v5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:50:14 -04:00
}
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
}
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
} // end direction (non-meta) calibration body
fix(deinit): vote arrows survived the cheap sweep, and 5 long loops ignored the stop Leftover chart objects on long-history charts. Two causes, one of them introduced by 07aa017. THE ONE I ADDED. The filtered view's overlay draws up to SIGNAL_RESCAN_LOOKBACK_BARS vote arrows. OnDeinit's EARLY VISIBLE-UI SWEEP runs with skipArrows=true, which skips any prefix equal to SIG_ARROW_PREFIX - and "WarSig_VOTE_..." starts with "WarSig_", so every one of them was skipped by the one sweep that is cheap enough to always complete. They then sat in the object list while the two expensive scans that follow walked it: a per-member SaveChartSignals O(total) scan, then the by-name rescan. On a chart with years of history that is thousands of extra objects walked twice, inside a teardown budget measured from the stop REQUEST rather than from OnDeinit's first line. skipArrows exists because the per-model arrows' sidecar is rebuilt by SCANNING them off the chart, so they cannot be deleted before that write. Vote arrows have no sidecar - they are a reconstruction, rebuilt on the next attach - so nothing is preserving them and they now get their own prefix slot, deleted by one native call in the first few milliseconds. THE FIVE LOOPS. A time budget bounds THROUGHPUT, not latency to an unload, and OnDeinit cannot begin until whatever is in flight returns. These all scaled with history and none of them checked: * Training passes 2, 2.5 and 3 yielded only on TRAIN_TIME_BUDGET_MS. Pass 1 has checked IsStopped() all along; the other three never have, and they are the ones that grow with the bar count. Free to fix - the resume state is written either way, so a stopped chunk simply is not re-entered. * PruneDirectionalClusters: the one UNCHUNKED sweep left, once per era over every bar, with its own header noting that raising the training budget cannot help its cost. Now bails outright. * AdvanceChartSignalRestore / AdvanceChartSignalRescan: chunked, but the rescan runs a full feedForward per bar over up to 5000 bars and the restore can hold MAX_RESTORED_ARROWS entries. Checked on the same 64-object stride as the clock read, since the check is not free either. * AdvanceFilteredOverlay (mine, 07aa017) replays Direction() on every classic filter per bar and had no check at all. Now per bar. ChartUI.mqh had ZERO shutdown checks across six loops before this. Nothing was added to the purge path itself: that is the work that must complete, and an IsStopped() check inside it would abort unconditionally - IsStopped() is already true by the time OnDeinit runs. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 17:13:50 -04:00
//--- Time OR stop - see pass 2's matching comment.
if(m_calibIndex - 1 >= calibLo && (IsStopped() || GetTickCount() - chunkStartTick >= TRAIN_TIME_BUDGET_MS))
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
{
//--- yield: m_isCalibActive + m_calibIndex carry the resume position, same as passes 1-3.
m_resumeBars = bars;
m_resumeTotalIter = totalIter;
m_resumeOosCutoff = oosCutoff;
m_resumeAddLoop = add_loop;
m_resumeBarIndex = i;
m_eraResumePending = true;
m_modelEta = eta;
return;
}
}
Net.SetBatchNormFrozen(false);
//--- An empty band (era too short to carve one - see CalibBandBars) means there is no measurement
//--- this era, which is not the same as a measurement that says "trade everything". Leave the
//--- operating point exactly where the last successful fit put it rather than refitting on nothing.
if(calibHi > calibLo)
FitDirConfThreshold();
m_isCalibActive = false;
m_isCalibDone = true;
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- Pass 3: OOS scoring, chronological, AFTER pass 2 has actually trained on this era's IS data -
//--- see m_isPass3Active's declaration comment for why this can no longer happen inline during
//--- pass 1's scan.
if(!stop && add_loop)
{
if(!m_isPass3Active)
{
m_isPass3Active = true;
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//--- Freeze batch-norm running statistics for the whole scoring walk (2026-08-09 audit, F5).
//--- Unfrozen, every scored bar advances the EMA mean/variance, so (a) the OOS number partly
//--- measures BN drift rather than the trained function, and (b) the same weights score
//--- differently depending on what was scored before them - and this pass produces the exact
//--- numbers checkpoint selection and the deploy gate rank on, which must be a pure function
//--- of the checkpoint. Same reasoning (and same mechanism) as ValidateCpuInference. The
//--- freeze persists across mid-pass chunk yields (the flag lives on the layers) and is
//--- lifted right after the walk completes; FinalizeTrainRun also unfreezes defensively for
//--- the stop-mid-pass path. Live/online adaptation is untouched - only scoring is frozen.
Net.SetBatchNormFrozen(true);
fix: drop the ranking slice for the calibration band; un-collapse the tiers NOT COMPILED - user compiles. (1) THE RANKING SLICE IS GONE. It reserved 20% of the OOS window so the pattern-DB backfill would read bars the deployed checkpoint was not SELECTED on. That objection stands; carving a new region to answer it did not. The calibration band already has every property the slice was buying: never trained on | never graded by pass 3 (which walks [0, oosCutoff) and so never reaches it) | never seen by the deploy gate | purged by a full label horizon on BOTH sides | and larger besides - 1,684 bars vs the ~970 carved So the backfill now walks [calibLo, calibHi) and pass 3 goes back to grading the entire OOS window, exactly as before any of this. The gate gets its full sample back (~10% of a sigma), the split loses a region, and the failure mode found an hour ago - a reserved region silently blanking ~10 months of chart arrows, because arrows are only drawn on bars pass 3 grades - becomes impossible. One impurity, stated in the completion log rather than hidden: m_dirConfThreshold is FITTED on that band and the walk applies it to decide which bars fired, so coverage there is mildly optimistic. One scalar under a coverage floor, against checkpoint selection over hundreds of eras. This backfill IS the deploy-time warm-up: it runs right after FinalizeTrainRun() restores the deployed weights, so it scores with exactly what is about to trade. (2) EVERY CALL WAS TIER 0, AND IT WAS ARITHMETIC. ConfidenceTier() quartiles [floorConf, 1] where floorConf = 1/3 - the lowest magnitude a 3-way softmax winner can hold. But it was fed CalibratedConfidenceMagnitude(), which multiplies by m_confidenceCalScale, clamped to [0.3, 1.5]. That lower clamp is BELOW 1/3. Whenever calibration bottoms out, t goes negative and MathMax(0, ...) pins every call to tier 0. Which is what the live run does. m_confidenceCalScale is EMA'd toward empiricalAccuracy / avgClaimedConfidence; with the model over-calling Neutral, 3-class agreement sits near 10% against a claimed confidence near 0.9, so the ratio is ~0.11 and clamps to 0.3 every era. Logged: tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0) 828 calls, one bucket - the four tier weights and the entire per-tier pattern-DB ranking reduced to a single number. The backfill was feeding a mechanism that structurally could not rank. Tiering now reads the RAW head magnitude, which genuinely lives on the [1/3, 1] range these bounds were written for. Calibration keeps its real jobs - AIConfidence() for MM sizing and SignedAIConfidence() for the vote are unchanged. STILL OPEN, deliberately not touched here: the calibration TARGET itself. empiricalAccuracy is 3-class agreement, which is the wrong quantity to scale a DIRECTIONAL confidence against - it counts a Neutral class that is 0.19% of labels. The honest target is the win rate on the calls the confidence describes (directional precision), with the claimed-confidence average taken over those same called bars. That needs a new accumulator and it interacts with the Neutral over-calling being fixed elsewhere, so it wants one clean run first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 23:06:11 -04:00
m_oosScoreStartIndex = (int)MathMin(oosCutoff - 1, bars - MathMax(m_historyBars, 0) - 2);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_oosScoreIndex = m_oosScoreStartIndex;
for(int rn = 0; rn < 3; rn++)
{
m_oosOutMin[rn] = DBL_MAX;
m_oosOutMax[rn] = -DBL_MAX;
}
m_oosOutSpreadSum = 0.0;
m_oosOutCount = 0;
feat(diagnostics): split a reported "Neutral" into CHOSE vs TIED - they need opposite fixes ApplyClassificationSoftmax() requires a STRICT majority over both rivals and sends every tie, 2-way or 3-way, to Neutral. So "OOS recall Neutral:100%" is two completely different events sharing one label: CHOSE - the net genuinely ranks Neutral highest. A class-prior/label problem. TIED - the top two are EXACTLY equal, so the net expressed no preference and the tie-break reported Neutral. A SATURATION problem: the head is SIGMOID, and a saturated sigmoid returns exactly 0.0f or 1.0f in the DLL's float32, so two classes pinned to the same rail compare equal and the bar is silently discarded. Nothing in the logs could tell them apart, and the fixes point opposite ways. Eras 1-25 of the 2026-08-17 solo PAI run read "Neutral 100%" at spread avg 0.99 - fully saturated - and broke out at era 27 as the spread fell to 0.75. That is consistent with EITHER story. The user reports the Neutral phase on most runs, so it is worth four longs to stop guessing. Four per-era counters on the pass 3 OOS walk, reported as: | Neutral CHOSE 12.4% / TIED 38.1% (of which B=S 1204) | rail 61.2% m_oosNeutralStrict - Neutral strictly highest m_oosNeutralTie - no strict winner; the tie-break produced Neutral m_oosTieBuySell - the costly subset: Buy and Sell tied AT the top, i.e. a DIRECTIONAL reading thrown away by float equality m_oosRailBars - any raw output sitting on a sigmoid asymptote, the saturation that makes exact ties possible at all Read on the RAW logits, before ApplyClassificationSoftmax() overwrites TempData in place. Legitimate because softmax is strictly monotone: it cannot change the ordering and cannot break a tie either, so the raw reading and the decision always agree. Placed alongside the existing min/max/spread capture so all the output diagnostics describe the same values. Measurement only - no decision path reads these. NOT COMPILED - user compiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:06:54 -04:00
m_oosNeutralStrict = 0;
m_oosNeutralTie = 0;
m_oosTieBuySell = 0;
m_oosRailBars = 0;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
fix(gate): move the ranking slice to the OLD end - it walled off the recent chart NOT COMPILED - user compiles. User: "there is quite some trading going on, but absolutely nothing on the recent area of the chart, like there is a hard wall starting around november 2025." That wall is 7caf2f6's ranking slice, and it was placed at the wrong end. Chart arrows are only ever drawn on bars pass 3 GRADES, and the slice reserved the NEWEST 20% of the OOS window plus a label-horizon purge. At the live sizing - ~4,860 OOS bars, 128-bar horizon - that is ~1,100 H4 bars withheld from grading, about ten months back from today, exactly where the wall appears. The invisible cost was worse than the visible one: it handed the deploy gate the OLDEST 80% of the OOS window and withheld the most recent regime from the single decision that has to generalise forward. Both fixed by putting the reserve at the oldest end instead: [0, oosScoreHi) OOS - graded by pass 3 (NEWEST, arrows restored) [oosScoreHi, rankLo) purge - one label horizon [rankLo, oosCutoff) RANKING - backfill only, graded by nobody [oosCutoff, calibLo) purge [calibLo, calibHi) CALIBRATION ... IS Of the three consumers competing for those bars, recency is worth least to the ranking: it is an ORDERING of confidence tiers, far less regime-sensitive than an absolute win rate, while the gate's power and the operator's read of the chart both want the newest data. The slice keeps every property that made it worth carving - never graded, never selected on, never seen by the gate, purged on both sides - so the backfilled rows are still honestly out-of-sample. RankSliceHiIndex is replaced by RankSliceLoIndex + OosScoreHiIndex; pass 3 now excludes the slice at the TOP of its walk and descends to 2 as it always did. The backfill walks [RankSliceLoIndex, oosCutoff) via a new m_dbBackfillStopIndex, clamped at both ends so a degenerate slice yields an empty walk rather than one that wanders into graded bars. Verified no reference to the old helper survives. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 22:58:55 -04:00
for(; m_oosScoreIndex >= 2; m_oosScoreIndex--)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
int oi = m_oosScoreIndex;
if(!(oi < (int)(bars - MathMax(m_historyBars, 0) - 1) && m_Time.GetData(oi) > dtStudied))
continue;
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
TrainHeartbeat("pass 3 (OOS scoring), bar", m_oosScoreStartIndex - m_oosScoreIndex + 1,
fix: the panel showed "100%" for the whole of pass 1 The simple panel derived its percentage from pass 2's counters: (m_isTrainCursor+1) / max(m_isTrainQueueCount,1). During pass 1 those are 0 and 0, so the expression is (0+1)/max(0,1) = 100%. An era spends its first pass scanning ~38k bars - minutes of work - and the panel reported that phase as finished the entire time. Observed by the user as "started learning at 100% of their era and are stuck there", and it actively misled the diagnosis: the one number on screen said the opposite of what was happening. The UI cannot fix this on its own - it can see pass 2's counters but has no way to know which pass owns them. So each pass now PUBLISHES its own progress and a short phase name through TrainHeartbeat (which every pass already calls per item), and the panel just displays them: "learning (era 45, scan 34%)". Published before the heartbeat's 4096-item journal gate, so the panel updates continuously while the journal stays quiet. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:22:29 -04:00
m_oosScoreStartIndex + 1, "scoring");
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- META TARGET OOS scoring, one row per candidate, feeding the SAME members the era-end
//--- selection/deploy block reads - under the win->Buy / loss->Sell mapping (pass 1 comment)
//--- every downstream figure keeps a correct meta meaning:
//--- dirPrecPct = wins among candidates the operating point trades (the win rate)
//--- chancePrec = base win rate of ALL candidates (always-call zero-skill reference,
//--- which under cost-charged win-counting IS the break-even coincidence
//--- the 2026-08-09 note below derives)
//--- coveragePct = fraction of candidates traded
//--- buy/sell recall = sensitivity/specificity, so bothSidesLive rejects the
//--- always-call and never-call collapses
//--- so checkpoint selection, the edge floor's standard error, the plateau ladder and the
//--- family-wise deploy gate all run UNCHANGED on the meta head.
if(IsMetaTarget())
{
for(int cd = MetaCandFirst(oi); cd >= 0; cd = MetaCandNext(cd))
{
ulong hbM = GetMicrosecondCount();
bool mWindowOk = BuildFeatureWindow(oi);
if(mWindowOk)
AppendCandidateFeatures(cd);
m_passFeatUs += GetMicrosecondCount() - hbM;
hbM = GetMicrosecondCount();
bool mForwardOk = (mWindowOk && TempData.Total() >= NetInputWidth() &&
Net.feedForward(TempData));
m_passNetUs += GetMicrosecondCount() - hbM;
if(!mForwardOk)
{
if(mWindowOk && !forwardFailureReported)
{
forwardFailureReported = true;
Print(__FUNCTION__ + ": CNet::feedForward FAILED during meta OOS scoring at era " +
IntegerToString((int)m_eraCount) + " - affected candidates are excluded.");
}
break;
}
Net.getResults(TempData);
double oPwin = MetaWinProbability();
bool oWon = MetaCandidateWon(cd, oi);
bool oCall = (oPwin >= 0.5); // the 2-class argmax
bool oHit = (oCall == oWon);
m_oosSamples++;
m_oosConfidenceSum += oPwin;
if(dOosError < 0)
dOosError = 0;
//--- mapped confusion counts (recall gate + balanced-accuracy diagnostics)
if(oWon)
{
m_oosBuyTotal++;
if(oHit)
m_oosBuyHits++;
//--- the always-call reference wins exactly when the candidate wins
m_oosWinLongTotal++;
}
else
{
m_oosSellTotal++;
if(oHit)
m_oosSellHits++;
}
//--- predicted-keyed tallies (panel Called/precision diagnostics)
if(oCall)
{
m_oosBuyPredicted++;
if(oHit)
m_oosBuyPredictedHits++;
if(oWon)
m_oosBuyPredictedWins++;
m_countBuySignals++;
//--- persistent OOS precision over called candidates, in WINS (matches the IS side)
m_cumOosTotal++;
if(oWon)
m_cumOosCorrect++;
}
else
{
m_oosSellPredicted++;
if(oHit)
m_oosSellPredictedHits++;
m_countSellSignals++;
}
//--- THE POPULATION THAT TRADES: candidates clearing the fitted operating point - what
//--- the deployability gate and selection score actually read (see the era-end block).
feat: per-family x per-side OOS breakdown in the META era report 350-era S2 verdict on SP500 H1: the meta head carries REAL ranking skill (+1.0-1.3pp mean over base, 101/350 eras clear their own 2-sigma bar, traded subset wins 66.1% at <30% coverage vs 64.5% base) but 0/350 eras produced a positive cov x (p - BE): the candidate stream sits 3pp under the derived geometry's 67.5% break-even and ~2.6pp of recovered skill cannot bridge it. Skill plateaued by mid-run (1.28pp -> 1.05pp), so more eras only buy multiplicity, and the deploy gate correctly shipped nothing. The aggregate can hide a deployable subset (one family/side clearing BE blended with junk), so the META era line now decomposes the SAME traded population into MA/RSI/MACD/Ichimoku x LONG/SHORT cells, each as traded/candidates base->traded win rate. 32 cells is a best-of-N search by construction - any candidate cell faces the family-wise rule before belief. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 09:03:40 -04:00
bool oFired = (oPwin >= m_dirConfThreshold);
if(oFired)
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
{
m_oosBuyFired++;
if(oWon)
m_oosBuyFiredHits++;
}
feat: per-family x per-side OOS breakdown in the META era report 350-era S2 verdict on SP500 H1: the meta head carries REAL ranking skill (+1.0-1.3pp mean over base, 101/350 eras clear their own 2-sigma bar, traded subset wins 66.1% at <30% coverage vs 64.5% base) but 0/350 eras produced a positive cov x (p - BE): the candidate stream sits 3pp under the derived geometry's 67.5% break-even and ~2.6pp of recovered skill cannot bridge it. Skill plateaued by mid-run (1.28pp -> 1.05pp), so more eras only buy multiplicity, and the deploy gate correctly shipped nothing. The aggregate can hide a deployable subset (one family/side clearing BE blended with junk), so the META era line now decomposes the SAME traded population into MA/RSI/MACD/Ichimoku x LONG/SHORT cells, each as traded/candidates base->traded win rate. 32 cells is a best-of-N search by construction - any candidate cell faces the family-wise rule before belief. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 09:03:40 -04:00
//--- per-family / per-side decomposition of the same population (see the declaration)
int oFam = m_metaCandFamily[cd];
int oSideIdx = (m_metaCandSide[cd] > 0) ? 0 : 1;
if(oFam >= 0 && oFam < 4)
{
m_metaFamCand[oFam]++;
if(oWon)
m_metaFamWins[oFam]++;
if(oFired)
{
m_metaFamFired[oFam]++;
if(oWon)
m_metaFamFiredWins[oFam]++;
}
}
m_metaSideCand[oSideIdx]++;
if(oWon)
m_metaSideWins[oSideIdx]++;
if(oFired)
{
m_metaSideFired[oSideIdx]++;
if(oWon)
m_metaSideFiredWins[oSideIdx]++;
}
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
if(oHit)
{
dOosForecast += (100 - dOosForecast) / Net.recentAverageSmoothingFactor;
dOosError -= dOosError / Net.recentAverageSmoothingFactor;
}
else
{
dOosForecast -= dOosForecast / Net.recentAverageSmoothingFactor;
dOosError += (100 - dOosError) / Net.recentAverageSmoothingFactor;
}
UpdateTrainingStatusLabel(
StringFormat("Scoring OOS bar %d of %d -> %.2f%% (meta)",
m_oosScoreStartIndex - m_oosScoreIndex + 1, m_oosScoreStartIndex + 1,
(double)(m_oosScoreStartIndex - m_oosScoreIndex + 1.0) /
MathMax(m_oosScoreStartIndex + 1, 1) * 100),
(TempData.Total() > 0) ? TempData[0] : 0.0,
(TempData.Total() > 1) ? TempData[1] : 0.0, 0.0, oPwin);
}
}
else
{
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
ulong hbT = GetMicrosecondCount();
fix: the sequence models were reading the window backwards BuildFeatureWindow() replaces eight hand-rolled copies of the same loop and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first, because MQL5 timeseries indices run backwards and `r + b` with b ascending walks into the past. Harmless for PAI and CONV - a dense layer learns a weight per position either way, a conv learns time-mirrored kernels. Not harmless for the recurrent stacks: - LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t. - It writes output[] only when t == steps-1: the visible output IS the last hidden state. - c_t = f*c_{t-1} + i*g decays toward the start of the sequence. lstm_seq_flowcheck.cpp measured block 0's influence on the output at 1.2e-2 of block T-1's, at the shipped forget bias of 1.0. So the bar being PREDICTED sat at the far end of the decay and the output was handed to the OLDEST bar in the window - the exact inverse of what the window is for. ~80x backwards on LSTM and HYBRID, on all three tiers (OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never surfaced as a backend discrepancy. This does not create edge - the MI diagnostics read at the noise floor (p=0.4975) with a working positive control. It makes the one hypothesis those diagnostics explicitly do NOT cover testable: they are marginal and per-bar, and state they "cannot rule out one that only exists in combination or across time". The sequence model is the instrument for across-time structure and it has been crippled, so that hypothesis has never been honestly tested. Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and its features, so a stale .nnw would load cleanly and run a model fitted to one ordering against the other, silently. Re-keying every config is the point, not collateral damage. FORCES A FULL RETRAIN. Also: the now-relative bar caches are re-keyed on the two live paths. EnsureBarCachesCapacity() was only ever called from training paths, but once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar to RefreshConvergedSignal() and Train() is never re-entered - so nothing cleared the feature cache again for the life of the process. A chart that trained to convergence kept replaying the rows computed for the last training era's bar grid: the live signal froze at its convergence-time value, and OnlineLearnStep() backpropped those stale features against freshly resolved labels. Backtests were never affected (an inference-only process never allocates the arrays, so every read recomputes). Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 18:28:44 -04:00
bool oWindowOk = BuildFeatureWindow(oi);
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
m_passFeatUs += GetMicrosecondCount() - hbT;
fix: improve error messages and suppress false sharing-violation logs - BufferDouble: replace hardcoded "DirectML/CPU-DLL" with dynamic backend name and add buffer index/element count to all error prints for easier debugging. - NetPersistence: distinguish missing file from transient lock by probing FileIsExist before logging, eliminating false "sharing violation" warnings when no saved model exists on first run.
2026-08-02 01:09:18 -04:00
//--- Same guard as pass 2, and it matters more here: OOS accuracy is what checkpoint selection
//--- and the plateau ladder's auto-deploy both rank on, so scoring a stale forward pass would
//--- not just be wrong, it would be wrong in the one number that decides which model ships.
//--- A skipped bar simply isn't counted; it never becomes a hit or a miss.
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
hbT = GetMicrosecondCount();
fix: improve error messages and suppress false sharing-violation logs - BufferDouble: replace hardcoded "DirectML/CPU-DLL" with dynamic backend name and add buffer index/element count to all error prints for easier debugging. - NetPersistence: distinguish missing file from transient lock by probing FileIsExist before logging, eliminating false "sharing violation" warnings when no saved model exists on first run.
2026-08-02 01:09:18 -04:00
bool oForwardOk = (oWindowOk && TempData.Total() >= (int)m_historyBars * m_neuronsCount &&
Net.feedForward(TempData));
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
m_passNetUs += GetMicrosecondCount() - hbT;
fix: improve error messages and suppress false sharing-violation logs - BufferDouble: replace hardcoded "DirectML/CPU-DLL" with dynamic backend name and add buffer index/element count to all error prints for easier debugging. - NetPersistence: distinguish missing file from transient lock by probing FileIsExist before logging, eliminating false "sharing violation" warnings when no saved model exists on first run.
2026-08-02 01:09:18 -04:00
if(oWindowOk && !oForwardOk && !forwardFailureReported)
{
forwardFailureReported = true;
Print(__FUNCTION__ + ": CNet::feedForward FAILED during OOS scoring at era " +
IntegerToString((int)m_eraCount) + " - affected bars are excluded from the OOS"
" accuracy rather than scored against a stale prediction.");
}
if(oForwardOk)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
feat: excursion-size head (Stage 1, measurement only) Direction is closed - normalised asymmetry fails on three instruments with a working positive control, and the classifier's own best-of-999 era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is a different question and RANGE clears at ~4x its null. Checked the denomination before building on that, since the source memo warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is predictable" is a claim about travel RELATIVE to current ATR, not a restatement of "ATR is autocorrelated". It is exactly the part a fixed multiple (stop 3.31*ATR, target 1.64*ATR) discards. A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches ladder rung k) upward and downward. Survival parameterisation rather than regressing the multiple, because it needs nothing new from CNet: sigmoid outputs and the per-neuron delta the `total != 3` branch already applies (a quantile head would need a linear activation and a pinball gradient in Network.mqh, Network.cl and the DirectML path, on a class four topologies share). Targets are free - m_ladderUpAt already records first-touch age per rung with 0 meaning never reached. Separate net, not extra outputs on the classifier: more outputs would change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push the count off 3 - the exact condition backProp uses to select the joint softmax gradient the 3-class head depends on. The classifier is bit-for-bit unaffected and this is removable without trace. STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the constant per-rung base rate - the baseline a fixed ATR multiple already assumes - with both predictors fitted IS and evaluated OOS, so neither gets a look at the test set. Positive skill justifies Stage 2 (drive SL/TP and sizing off ExcursionQuantile, which is defined and deliberately uncalled). Zero or negative means ATR already carries everything and Stage 2 must not be built. Trains only on primary occurrences: the replay queue oversamples for CLASS balance, and a direction-balanced sample is a biased SIZE sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
//--- EXCURSION HEAD scored on the SAME held-out bars the classifier is graded on, and for
//--- the same reason: it never trained on them. Before getResults() overwrites TempData.
ExcursionScoreStep(oi);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
Net.getResults(TempData);
// Raw output stats MUST be captured here, before ApplyClassificationSoftmax() overwrites
// TempData[0..2] in place with the softmax probabilities - see m_oosOutMin's declaration
// comment for what these feed.
if(m_outputNeuronsCount == 3 && TempData.Total() >= 3)
{
double rawHi = -DBL_MAX, rawLo = DBL_MAX;
for(int rn = 0; rn < 3; rn++)
{
double rv = TempData.At(rn);
if(rv < m_oosOutMin[rn])
m_oosOutMin[rn] = rv;
if(rv > m_oosOutMax[rn])
m_oosOutMax[rn] = rv;
rawHi = MathMax(rawHi, rv);
rawLo = MathMin(rawLo, rv);
}
m_oosOutSpreadSum += rawHi - rawLo;
m_oosOutCount++;
feat(diagnostics): split a reported "Neutral" into CHOSE vs TIED - they need opposite fixes ApplyClassificationSoftmax() requires a STRICT majority over both rivals and sends every tie, 2-way or 3-way, to Neutral. So "OOS recall Neutral:100%" is two completely different events sharing one label: CHOSE - the net genuinely ranks Neutral highest. A class-prior/label problem. TIED - the top two are EXACTLY equal, so the net expressed no preference and the tie-break reported Neutral. A SATURATION problem: the head is SIGMOID, and a saturated sigmoid returns exactly 0.0f or 1.0f in the DLL's float32, so two classes pinned to the same rail compare equal and the bar is silently discarded. Nothing in the logs could tell them apart, and the fixes point opposite ways. Eras 1-25 of the 2026-08-17 solo PAI run read "Neutral 100%" at spread avg 0.99 - fully saturated - and broke out at era 27 as the spread fell to 0.75. That is consistent with EITHER story. The user reports the Neutral phase on most runs, so it is worth four longs to stop guessing. Four per-era counters on the pass 3 OOS walk, reported as: | Neutral CHOSE 12.4% / TIED 38.1% (of which B=S 1204) | rail 61.2% m_oosNeutralStrict - Neutral strictly highest m_oosNeutralTie - no strict winner; the tie-break produced Neutral m_oosTieBuySell - the costly subset: Buy and Sell tied AT the top, i.e. a DIRECTIONAL reading thrown away by float equality m_oosRailBars - any raw output sitting on a sigmoid asymptote, the saturation that makes exact ties possible at all Read on the RAW logits, before ApplyClassificationSoftmax() overwrites TempData in place. Legitimate because softmax is strictly monotone: it cannot change the ordering and cannot break a tie either, so the raw reading and the decision always agree. Placed alongside the existing min/max/spread capture so all the output diagnostics describe the same values. Measurement only - no decision path reads these. NOT COMPILED - user compiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:06:54 -04:00
//--- WHY Neutral won, split into its two causes - see m_oosNeutralStrict's declaration.
//--- Read on the RAW logits, which is legitimate because the softmax below is strictly
//--- monotone: it cannot change the ordering, and it cannot break a tie either. Doing it
//--- here also means these counters see the same values the min/max/spread stats do,
//--- BEFORE ApplyClassificationSoftmax() overwrites TempData[0..2] in place.
double rB = TempData.At(0), rS = TempData.At(1), rN = TempData.At(2);
bool strictB = (rB > rS && rB > rN);
bool strictS = (rS > rB && rS > rN);
bool strictN = (rN > rB && rN > rS);
if(strictN)
m_oosNeutralStrict++;
else
if(!strictB && !strictS)
{
//--- No class holds a strict majority, so the top two are EXACTLY equal and
//--- ApplyClassificationSoftmax() returns Neutral by the tie rule, not by choice.
m_oosNeutralTie++;
//--- The expensive subset: Buy and Sell tied AT the top (either a 2-way tie above
//--- Neutral, or a 3-way). The net had a directional reading and float equality
//--- threw it away.
if(rB == rS && rB >= rN)
m_oosTieBuySell++;
}
//--- Sigmoid rails. The head is SIGMOID (Topology.mqh), so 0 and 1 are its asymptotes;
//--- a raw value sitting ON one in float32 is the saturation that MAKES exact ties
//--- possible. Counted per BAR, not per output, so this is directly comparable to
//--- m_oosOutCount.
if(rawLo <= 1e-6 || rawHi >= 1.0 - 1e-6)
m_oosRailBars++;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
double oPrevSignal = (m_outputNeuronsCount == 3) ? ApplyClassificationSoftmax() : TempData[0];
double oDeploySignal = oPrevSignal;
if(m_outputNeuronsCount == 3)
oDeploySignal = AdjustedSignalFromSoftmax();
double oNeuron0 = (TempData.Total() > 0) ? TempData[0] : 0.0;
double oNeuron1 = (TempData.Total() > 1) ? TempData[1] : 0.0;
double oNeuron2 = (TempData.Total() > 2) ? TempData[2] : 0.0;
bool oBuy = m_labelCacheHasValue[oi] ? m_labelCacheBuy[oi] : false;
bool oSell = m_labelCacheHasValue[oi] ? m_labelCacheSell[oi] : false;
ENUM_SIGNAL oTrueSignal = oBuy ? Buy : (oSell ? Sell : Neutral);
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
//--- Per-direction OUTCOMES, kept apart from the label - see m_oosBuyPredictedWins.
bool oWinLong = (m_labelCacheHasValue[oi] && oi < ArraySize(m_winLongCache))
? m_winLongCache[oi] : false;
bool oWinShort = (m_labelCacheHasValue[oi] && oi < ArraySize(m_winShortCache))
? m_winShortCache[oi] : false;
feat(vote): thresholds become confidence percentages, on ONE scale everywhere User request: "the entry/exit thresholds are manual numbers, I would like them to be confidence percentages, so the current 20 would be only 20% confidence in a profitable trade." WHY 20 WAS EVER SENSIBLE. Under UseDatabaseRanking both factors of a filter's contribution are win rates: the pattern weight is that pattern's measured win rate (UpdateSignalsWeights -> ApplyPatternWeight) and m_weight is the filter's average win rate over its patterns, /100. Dividing the sum by the VOTER COUNT therefore produced a mean of PRODUCTS of two win rates - a genuinely 60%-accurate filter firing a 60% pattern scored 0.60 x 60 = 36. The number was never on a probability scale, so its magnitude meant nothing on its own. Dividing by Sum(m_weight) instead makes it a weighted MEAN of win rates, which is a win rate: result = Sum(w_i*p_i)/Sum(w_i). Every voter at 60% now reads 60; MACD's double-divergence pattern (weight 100) voting alone reads 100. m_weight stops being a discount on the probability and becomes how much a filter's opinion COUNTS - which is what a module weight should always have been. Default Min_Vote_Open 20 -> 50: not a tightening, the same bar re-expressed. ONE SCALE, EVERYWHERE - the part that made this bigger than a rescale. Three other places compared against a 0..1 softmax confidence and would each have become a fresh currency mismatch the moment the input changed meaning: * the AI early-exit route (LiveSignedConfidence vs m_ai_exit_threshold) now reads m_lastAiVote - the AI filters' own weighted mean, undiluted by the classic side, which is the only reason that route exists - against the same m_threshold_close the averaged vote uses. m_ai_exit_threshold is retired rather than left dangling. * m_oosDecisionSeries now carries the vote, not the confidence, so the exit SIMULATION stops modelling a close rule the EA does not run. * ExitPolicy() clamped anything > 1.0 to zero. Passing the unscaled input through that would have silently switched vote exits off in the simulation while live went on running them - found before it shipped; the bound now tracks the scale. LiveSignedConfidence() is deliberately untouched and still 0..1: MM sizing, SL/TP scaling and the intelligent trailing want a model confidence, not a win rate. CALIBRATION CAVEAT, stated in the code where the claim is made: this is only a real probability to the extent the pattern weights are. A pattern with fewer than MIN_TRADES_FOR_WIN_RATE journaled trades keeps its DEFAULT weight - a designed prior (25/50/75/100 for the AI tiers), not a measurement. Until the signal DB fills, "60" means "the designed conviction of the patterns that fired". Closing that gap is the next commit. Also corrects VOTE_CLOSE_PRESETS' comment, which documented the two scales this removes. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:52:08 -04:00
//--- ONE CURRENCY, and it is the live one. oEnsembleVote is the signed vote this member
//--- would have cast on this bar - m_weight x its tier's pattern weight, the same number
//--- CExpertSignalCustom::Direction() sums and the same 0-100 win-rate scale
//--- Min_Vote_Open/Min_Vote_Close are expressed in. It feeds BOTH consumers below:
//--- * the ensemble deploy gate, via EnsembleOosContribute;
//--- * the exit simulation, via m_oosDecisionSeries.
fix(gate): the ensemble gate certified a vote the EA never casts g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on the same criterion the live trade does". It did not. Two independent mismatches, both silent: CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100 number straight off the softmax head. Live contributes m_weight x the tier's pattern weight, and BOTH of those are rewritten from the signal DB by UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share an axis and nothing relates them, so the same bar was one number to the gate and a different one to the order path. Same shape as the 2026-08-09 geometry incident: certified on one game, paid on another. DENOMINATOR. The gate divided by the member count, so an abstaining member pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero contribution in BOTH the sum and the count (`if(direction == 0) continue;` before `number++`) - live is a mean over VOTERS. The gate was therefore scoring a strictly more agreement-heavy set of bars than the EA trades. The contribution hook's own comment asserted the opposite ("abstentions dilute the average exactly as they do in the live vote"), while the AI_CHOICE enum 20 lines away correctly documented union semantics. LiveVoteContribution() is now the single definition of "what this member votes", called from the gate; the live path reaches the same arithmetic through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually voted, separately from who evaluated the bar, because those are the divisor and the shared-population test respectively. ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar wrapper - the OOS scan holds the scanned bar's decision in a local, and dPrevSignal is a different bar. NOT changed, deliberately: the gate still does not model live NMS declustering, and the per-member solo gate still scores every directional call rather than threshold-clearing ones. Both are selection-metric changes and this codebase has twice been bitten by switching one blind. Also corrects a stale paragraph in m_pattern_0's declaration block quoting 80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has been since the confidence floor and alternation gate were removed; the block carried both tables at once, and the dead one was quoted back as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
//---
feat(vote): thresholds become confidence percentages, on ONE scale everywhere User request: "the entry/exit thresholds are manual numbers, I would like them to be confidence percentages, so the current 20 would be only 20% confidence in a profitable trade." WHY 20 WAS EVER SENSIBLE. Under UseDatabaseRanking both factors of a filter's contribution are win rates: the pattern weight is that pattern's measured win rate (UpdateSignalsWeights -> ApplyPatternWeight) and m_weight is the filter's average win rate over its patterns, /100. Dividing the sum by the VOTER COUNT therefore produced a mean of PRODUCTS of two win rates - a genuinely 60%-accurate filter firing a 60% pattern scored 0.60 x 60 = 36. The number was never on a probability scale, so its magnitude meant nothing on its own. Dividing by Sum(m_weight) instead makes it a weighted MEAN of win rates, which is a win rate: result = Sum(w_i*p_i)/Sum(w_i). Every voter at 60% now reads 60; MACD's double-divergence pattern (weight 100) voting alone reads 100. m_weight stops being a discount on the probability and becomes how much a filter's opinion COUNTS - which is what a module weight should always have been. Default Min_Vote_Open 20 -> 50: not a tightening, the same bar re-expressed. ONE SCALE, EVERYWHERE - the part that made this bigger than a rescale. Three other places compared against a 0..1 softmax confidence and would each have become a fresh currency mismatch the moment the input changed meaning: * the AI early-exit route (LiveSignedConfidence vs m_ai_exit_threshold) now reads m_lastAiVote - the AI filters' own weighted mean, undiluted by the classic side, which is the only reason that route exists - against the same m_threshold_close the averaged vote uses. m_ai_exit_threshold is retired rather than left dangling. * m_oosDecisionSeries now carries the vote, not the confidence, so the exit SIMULATION stops modelling a close rule the EA does not run. * ExitPolicy() clamped anything > 1.0 to zero. Passing the unscaled input through that would have silently switched vote exits off in the simulation while live went on running them - found before it shipped; the bound now tracks the scale. LiveSignedConfidence() is deliberately untouched and still 0..1: MM sizing, SL/TP scaling and the intelligent trailing want a model confidence, not a win rate. CALIBRATION CAVEAT, stated in the code where the claim is made: this is only a real probability to the extent the pattern weights are. A pattern with fewer than MIN_TRADES_FOR_WIN_RATE journaled trades keeps its DEFAULT weight - a designed prior (25/50/75/100 for the AI tiers), not a measurement. Until the signal DB fills, "60" means "the designed conviction of the patterns that fired". Closing that gap is the next commit. Also corrects VOTE_CLOSE_PRESETS' comment, which documented the two scales this removes. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:52:08 -04:00
//--- Both used to be handed the raw signed CONFIDENCE instead (the gate then multiplying it
//--- by 100), which is a plausible-looking number in the right RANGE and the wrong
//--- CURRENCY: confidence is a head output, the live contribution is a DB-ranked win-rate
//--- weight, and nothing ties them together. Certifying on one while trading the other is
//--- the 2026-08-09 geometry incident's exact shape, and the exit simulation was modelling
//--- a close rule the EA does not run.
fix(gate): the ensemble gate certified a vote the EA never casts g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on the same criterion the live trade does". It did not. Two independent mismatches, both silent: CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100 number straight off the softmax head. Live contributes m_weight x the tier's pattern weight, and BOTH of those are rewritten from the signal DB by UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share an axis and nothing relates them, so the same bar was one number to the gate and a different one to the order path. Same shape as the 2026-08-09 geometry incident: certified on one game, paid on another. DENOMINATOR. The gate divided by the member count, so an abstaining member pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero contribution in BOTH the sum and the count (`if(direction == 0) continue;` before `number++`) - live is a mean over VOTERS. The gate was therefore scoring a strictly more agreement-heavy set of bars than the EA trades. The contribution hook's own comment asserted the opposite ("abstentions dilute the average exactly as they do in the live vote"), while the AI_CHOICE enum 20 lines away correctly documented union semantics. LiveVoteContribution() is now the single definition of "what this member votes", called from the gate; the live path reaches the same arithmetic through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually voted, separately from who evaluated the bar, because those are the divisor and the shared-population test respectively. ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar wrapper - the OOS scan holds the scanned bar's decision in a local, and dPrevSignal is a different bar. NOT changed, deliberately: the gate still does not model live NMS declustering, and the per-member solo gate still scores every directional call rather than threshold-clearing ones. Both are selection-metric changes and this codebase has twice been bitten by switching one blind. Also corrects a stale paragraph in m_pattern_0's declaration block quoting 80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has been since the confidence floor and alternation gate were removed; the block carried both tables at once, and the dead one was quoted back as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
double oEnsembleVote = LiveVoteContribution(oDeploySignal);
feat(vote): CONSENSUS arithmetic - agreement is now what the threshold dials Era-680 report, all three observations one equation: "peak 29, no arrows at threshold 30" / "at 20, arrows on EVERY bar" / "label at 12 while arrows everywhere". Under the voters-only divisor, any bar with at least one directional voter read the weighted mean of the firing tiers' weights - and once the tiers self-ranked to each model's pooled win rate (~28-31), that mean was NEAR-CONSTANT regardless of headcount. One member alone: ~29. Four unanimous: ~29. Min_Vote_Open was a step function around that constant - above it nothing ever fired, below it everything did - and the label's 12 was a 3v1 split netting through the same divisor. Not three display bugs: one arithmetic that could not express agreement. The divisor is now the CAPABLE weight - every filter that could vote, whether it did or not: * live (Direction): VoteCapableWeight() - classic pattern ladders always, veto filters never, AI members once past the same readiness test LongCondition gates on. A model still training must not dilute an ensemble it cannot join: four trainees + one deployed model is a solo chart wearing an ensemble label, and the solo vote reads full strength. * gate (EnsembleEraVerdict): g_ensVoteWeightSum accumulates for every member that EVALUATED the bar, Neutral included. * overlay sweep + prospective readout: weight counts whenever the member has data; a snapshotted Neutral dilutes. One arithmetic, four sites, same numbers everywhere. What the numbers become (four members, w~0.29, tiers~29): unanimous ~29 - the CEILING, which is the pooled win rate and is what the peak displays; 3-of-4 ~22; 2-of-4 ~14.5; 3v1 ~14.5. Min_Vote_Open 20 now means "roughly three-quarters of the ensemble's trust agrees, net". It MUST sit below the ceiling to ever fire - the census/peak states the ceiling. This is the ensemble the user specified in the original design discussion ("if the perceptron also votes, both together reach the threshold; if another NN votes the other side, the threshold is not reached") - union semantics was the pre-ensemble behaviour, kept until measurement showed its vote magnitude was a constant. Plus overlay DECLUSTERING, the other half of "arrows on every bar": the same three NMS rules as the per-member arrows (same-direction runs collapse to their first bar, cross-direction flicker keeps the stronger side), online over the sweep's strictly oldest->newest walk. Suppression is a verdict and deletes a standing arrow; the den==0 no-data skip still never does. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 06:58:03 -04:00
//--- The divisor term that goes with it - the member's weight WHENEVER it evaluated the
//--- bar, Neutral included, because consensus arithmetic (2026-08-19) has abstention
//--- dilute. Was zeroed on abstention under union semantics.
double oEnsembleWeight = ModuleWeight();
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
if(m_ensembleMember && m_labelCacheHasValue[oi])
feat(vote): thresholds become confidence percentages, on ONE scale everywhere User request: "the entry/exit thresholds are manual numbers, I would like them to be confidence percentages, so the current 20 would be only 20% confidence in a profitable trade." WHY 20 WAS EVER SENSIBLE. Under UseDatabaseRanking both factors of a filter's contribution are win rates: the pattern weight is that pattern's measured win rate (UpdateSignalsWeights -> ApplyPatternWeight) and m_weight is the filter's average win rate over its patterns, /100. Dividing the sum by the VOTER COUNT therefore produced a mean of PRODUCTS of two win rates - a genuinely 60%-accurate filter firing a 60% pattern scored 0.60 x 60 = 36. The number was never on a probability scale, so its magnitude meant nothing on its own. Dividing by Sum(m_weight) instead makes it a weighted MEAN of win rates, which is a win rate: result = Sum(w_i*p_i)/Sum(w_i). Every voter at 60% now reads 60; MACD's double-divergence pattern (weight 100) voting alone reads 100. m_weight stops being a discount on the probability and becomes how much a filter's opinion COUNTS - which is what a module weight should always have been. Default Min_Vote_Open 20 -> 50: not a tightening, the same bar re-expressed. ONE SCALE, EVERYWHERE - the part that made this bigger than a rescale. Three other places compared against a 0..1 softmax confidence and would each have become a fresh currency mismatch the moment the input changed meaning: * the AI early-exit route (LiveSignedConfidence vs m_ai_exit_threshold) now reads m_lastAiVote - the AI filters' own weighted mean, undiluted by the classic side, which is the only reason that route exists - against the same m_threshold_close the averaged vote uses. m_ai_exit_threshold is retired rather than left dangling. * m_oosDecisionSeries now carries the vote, not the confidence, so the exit SIMULATION stops modelling a close rule the EA does not run. * ExitPolicy() clamped anything > 1.0 to zero. Passing the unscaled input through that would have silently switched vote exits off in the simulation while live went on running them - found before it shipped; the bound now tracks the scale. LiveSignedConfidence() is deliberately untouched and still 0..1: MM sizing, SL/TP scaling and the intelligent trailing want a model confidence, not a win rate. CALIBRATION CAVEAT, stated in the code where the claim is made: this is only a real probability to the extent the pattern weights are. A pattern with fewer than MIN_TRADES_FOR_WIN_RATE journaled trades keeps its DEFAULT weight - a designed prior (25/50/75/100 for the AI tiers), not a measurement. Until the signal DB fills, "60" means "the designed conviction of the patterns that fired". Closing that gap is the next commit. Also corrects VOTE_CLOSE_PRESETS' comment, which documented the two scales this removes. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:52:08 -04:00
EnsembleOosContribute(oi, oEnsembleVote, oEnsembleWeight, oWinLong, oWinShort, (oBuy || oSell));
feat(gate): grade OOS calls on the exit policy actually in force, and move vote combining out of the members and into the orchestrator Option (a) from the exit-policy question: the certified number must be the traded number. Plus the modularity correction the user called for on 778b6c0. 1. VOTE COMBINING BELONGS TO THE ORCHESTRATOR, NOT TO A MEMBER. 778b6c0 fixed the last-writer-wins bug on g_LiveAISignedConfidence by having a member average its siblings through g_warriorEnsemble. That trades a scheduling bug for a coupling bug, and it is the wrong shape for this EA: every signal runs in its own instance, minds its own state, and VOTES to the orchestrator, which is the only thing allowed to combine opinions. Replaced with a publish/aggregate pair in Variables\ConfidenceBridge.mqh whose split is enforced by shape rather than by convention: - PublishAIVote(slot, conf) - a member writes ONLY its own slot, reads nobody's; - AggregateAIVotes() - called by CExpertSignalCustom::LiveSignedConfidence. CExpertSignalAIBase::EnsembleLiveSignedConfidence is gone. The orchestrator also republishes the aggregate into g_LiveAISignedConfidence, because the intelligent trailing reads that global directly and must act on the same number the exit route does rather than on a leftover from whichever member ticked last. A solo AI signal owns slot 0, so the non-ensemble path is unchanged. 2. THE GATE NOW REPLAYS THE REAL EXIT RULE. SimulateTradeOutcome() walks the same price series with the same fill/barrier/spread convention as ComputeLabelForBar - deliberately by copy, so a disagreement between the two can only be a policy effect and never a discrepancy between two pieces of our own arithmetic - and terminates at the FIRST of stop / target / vote reversal / horizon. Barriers are tested before the vote on the same bar: intrabar we cannot know which came first, and the barrier is what the broker executes automatically, so checking the vote first would credit the exit policy with escapes a real stop would have taken out of its hands. It runs AFTER pass 3, not inside it. A vote-flip exit for a trade entered at bar r is decided by the model's output at bars r-1, r-2, ... - NEWER bars - and pass 3 walks oldest-to-newest, so at the moment r is graded its own exit does not exist yet. Only once m_oosDecisionSeries is complete over the whole OOS window can the replay run. In ensemble mode that series carries the member's adjusted decision and the live exit reads the ensemble aggregate, which is the coupling the user identified: an LSTM entry really can be closed by the ensemble turning against it. 3. THE STATISTIC HAS TO CHANGE WITH THE POLICY, AND THAT IS THE REAL FINDING. A barrier exit pays a fixed R. A vote exit pays whatever the close happens to be. So the moment vote exits are enabled the payoff is CONTINUOUS, and "win rate vs break-even" stops being a meaningful test - there is no fixed break-even for a variable payoff. SimulateTradeOutcome therefore returns R rather than a bool, and the replay reports expectancy in R with its SE taken from the R distribution (overlap- deflated on the same EffectiveSampleSize doctrine as every other SE here), not from a binomial. This is the same class of error as win-based scoring in 2026-08-09: measuring a variable-payoff process with a fixed-payoff statistic. Naming it now, while vote exits are still off, is much cheaper than discovering it after they go on. 4. WHY THIS IS SAFE TO SHIP TODAY. Min_Vote_Close ships Disabled (101, unreachable on both scales it drives) and reaches the AI signal through the new ExitPolicy() setter as 1.01, which the setter turns into 0.0 = no vote exit. Under that policy every replayed trade resolves at a barrier and the simulation is arithmetically the same trade the deploy gate already certifies - they cannot drift. The report says so explicitly, and prints ONCE per run in that state; when vote exits are on it prints every era, because then the divergence is the thing to watch. Nothing about today's numbers moves. The gate switchover is wired but dormant by construction: it becomes exit-aware the moment the input is enabled, which is exactly what "the certified number is the traded number" has to mean. KNOWN LIMIT, stated rather than hidden: only the AI early-exit route is replayed. The rule-based averaged-vote close (m_threshold_close) depends on every other filter's live vote, which pass 3 does not reproduce, so a position the classic filters would have closed is held to its barrier here. The replay therefore holds LONGER than live and overstates barrier-reached outcomes. Faithful only while the AI is the dominant vote - which is the configuration this is being built for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:25:57 -04:00
//--- THE DECISION SERIES, for the exit simulation. Recorded for EVERY scanned OOS bar, not
//--- only the ones that fire, because a vote-flip exit is read at bars the model did NOT
//--- enter on - it is the reversal that closes a position opened earlier. Stored on the
//--- bar's own series index so SimulateTradeOutcome can walk it forward against price.
if(oi >= 0 && oi < ArraySize(m_oosDecisionSeries))
feat(vote): thresholds become confidence percentages, on ONE scale everywhere User request: "the entry/exit thresholds are manual numbers, I would like them to be confidence percentages, so the current 20 would be only 20% confidence in a profitable trade." WHY 20 WAS EVER SENSIBLE. Under UseDatabaseRanking both factors of a filter's contribution are win rates: the pattern weight is that pattern's measured win rate (UpdateSignalsWeights -> ApplyPatternWeight) and m_weight is the filter's average win rate over its patterns, /100. Dividing the sum by the VOTER COUNT therefore produced a mean of PRODUCTS of two win rates - a genuinely 60%-accurate filter firing a 60% pattern scored 0.60 x 60 = 36. The number was never on a probability scale, so its magnitude meant nothing on its own. Dividing by Sum(m_weight) instead makes it a weighted MEAN of win rates, which is a win rate: result = Sum(w_i*p_i)/Sum(w_i). Every voter at 60% now reads 60; MACD's double-divergence pattern (weight 100) voting alone reads 100. m_weight stops being a discount on the probability and becomes how much a filter's opinion COUNTS - which is what a module weight should always have been. Default Min_Vote_Open 20 -> 50: not a tightening, the same bar re-expressed. ONE SCALE, EVERYWHERE - the part that made this bigger than a rescale. Three other places compared against a 0..1 softmax confidence and would each have become a fresh currency mismatch the moment the input changed meaning: * the AI early-exit route (LiveSignedConfidence vs m_ai_exit_threshold) now reads m_lastAiVote - the AI filters' own weighted mean, undiluted by the classic side, which is the only reason that route exists - against the same m_threshold_close the averaged vote uses. m_ai_exit_threshold is retired rather than left dangling. * m_oosDecisionSeries now carries the vote, not the confidence, so the exit SIMULATION stops modelling a close rule the EA does not run. * ExitPolicy() clamped anything > 1.0 to zero. Passing the unscaled input through that would have silently switched vote exits off in the simulation while live went on running them - found before it shipped; the bound now tracks the scale. LiveSignedConfidence() is deliberately untouched and still 0..1: MM sizing, SL/TP scaling and the intelligent trailing want a model confidence, not a win rate. CALIBRATION CAVEAT, stated in the code where the claim is made: this is only a real probability to the extent the pattern weights are. A pattern with fewer than MIN_TRADES_FOR_WIN_RATE journaled trades keeps its DEFAULT weight - a designed prior (25/50/75/100 for the AI tiers), not a measurement. Until the signal DB fills, "60" means "the designed conviction of the patterns that fired". Closing that gap is the next commit. Also corrects VOTE_CLOSE_PRESETS' comment, which documented the two scales this removes. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:52:08 -04:00
m_oosDecisionSeries[oi] = oEnsembleVote;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
UpdateTrainingStatusLabel(
StringFormat("Scoring OOS bar %d of %d -> %.2f%% (post-training)", m_oosScoreStartIndex - m_oosScoreIndex + 1, m_oosScoreStartIndex + 1,
(double)(m_oosScoreStartIndex - m_oosScoreIndex + 1.0) / MathMax(m_oosScoreStartIndex + 1, 1) * 100),
oNeuron0, oNeuron1, oNeuron2, oDeploySignal);
// Held-out bar: score the model's freshly-trained-this-era forecast against the actual
// outcome without learning from it - keeps the OOS accuracy an honest overfitting signal.
bool oClassified = (DoubleToSignal(oPrevSignal) == Buy || DoubleToSignal(oPrevSignal) == Sell || DoubleToSignal(oPrevSignal) == Neutral);
if(oClassified)
{
m_oosSamples++;
m_oosConfidenceSum += MathAbs(oPrevSignal);
if(dOosError < 0)
dOosError = 0;
bool hit = (DoubleToSignal(oPrevSignal) == oTrueSignal);
ENUM_SIGNAL oPred = DoubleToSignal(oPrevSignal);
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
//--- Did the TRADE this call implies actually pay? Distinct from `hit`, which asks the
//--- narrower question of whether the call matched the single label the bar was collapsed
//--- to. On a both-won bar the label names one direction and this pays either way.
bool oTradeWon = (oPred == Buy) ? oWinLong : ((oPred == Sell) ? oWinShort : false);
//--- Zero-skill reference, measured over EVERY scored bar (not just the called ones):
//--- what always-long and always-short would have collected. See m_oosWinLongTotal.
if(oWinLong)
m_oosWinLongTotal++;
if(oWinShort)
m_oosWinShortTotal++;
//--- Compounded, persistent DIRECTIONAL win-rate: count only bars the model actually called
//--- Buy or Sell (Neutral "no trade" calls aren't wins or losses). Scored on oTradeWon, so
//--- the number the panel shows under "win rate" is one - it used to be label agreement,
//--- which is a different quantity and reads low by exactly the both-won share.
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
if(oPred == Buy || oPred == Sell)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
m_cumOosTotal++;
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
if(oTradeWon)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_cumOosCorrect++;
}
// Per-class confusion counts, used for the Buy/Sell recall convergence gate below
switch(oTrueSignal)
{
case Buy:
m_oosBuyTotal++;
if(hit)
m_oosBuyHits++;
break;
case Sell:
m_oosSellTotal++;
if(hit)
m_oosSellHits++;
break;
default:
m_oosNeutralTotal++;
if(hit)
m_oosNeutralHits++;
break;
}
fix: NMS gates the TRADE, not just the arrow - one arrow is now one trade NmsLiveAccept() appeared in exactly one place: wrapped around DrawObject(). It never touched dPrevSignal, and dPrevSignal is what LongCondition() / ShortCondition() / SignedAIConfidence() read. So a declustered bar lost its arrow and still opened a position. Measured on SP500 H1 2026-08-09: CONV called a direction on 64% of bars, so the ~500 bars visible on screen held ~320 decisions - and ~40 arrows were drawn. Roughly one arrow per eight positions the EA would take. And the survivors are not a random eighth. Rule 2 of the declustering keeps the HIGHER-CONFIDENCE side of a cluster, so the visible set is systematically the best member of each run. A chart showing the best of every eight decisions and hiding the rest reads far better than the model is - the same best-of-N selection error already corrected in the geometry scan, the indicator tuner, the lag profile and the deploy gate, this time on the display layer, where it is most likely to mislead the person deciding whether to trade. Fixed by neutralising dPrevSignal when NMS rejects, rather than adding a "may trade" flag consulted at each read site: that leaves exactly ONE definition of what the model decided this bar, so the arrow, the panel's "Current signal", the confidence feeding sizing/SL/TP/trailing, the refresh tally and the order itself cannot drift apart again. Also reports the consequence instead of hiding it. Every OOS counter on the era line still scores every directional call - a population ~8x larger than what now trades - so the line carries a second figure: | TRADED (declustered) NN% on N calls (edge +Npp) replaying the identical rule over pass 3 (which walks OOS bars oldest to newest, the same order the live sweep sees). Its cursors are separate members from the live ones so a training pass can never disturb the live chart's declustering. Deliberately NOT switched into selectionScore yet. Declustering cuts coverage from ~64% of bars to ~8%, well under MIN_COVERAGE_FRACTION_OF_BASE_RATE, which would make every checkpoint undeployable overnight - the minRR collision and the recall-floor catch-22 twice over. The floor gets re-derived from these measurements first. Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-09 10:22:31 -04:00
//--- DECLUSTERED count: of the calls that would actually become POSITIONS, how many were
//--- right. Since 2026-08-09 live NMS gates the trade and not just the arrow (see
//--- RefreshLatestSignal), so every other figure on this line describes a strictly larger
//--- population than the EA trades - roughly 8x larger at the shipped 6-bar window. This
//--- pair is the one that answers "what would I have made".
//--- Same rule as PruneDirectionalClusters/NmsLiveAccept, replayed here because pass 3
//--- walks OOS bars oldest-to-newest (m_oosScoreIndex descends, and a HIGH index is an OLD
//--- bar), which is exactly the order the live sweep sees them in.
//--- Reported alongside, NOT substituted into selectionScore: declustering cuts coverage
//--- from ~64% of bars to ~8%, which sits below MIN_COVERAGE_FRACTION_OF_BASE_RATE and
//--- would make every checkpoint undeployable overnight. That is the minRR and recall-floor
//--- catch-22 twice over, so the floor gets re-derived from these measurements first.
if(m_signalClusterWindow > 0)
{
fix: the deploy gate graded the un-thresholded model coveragePct, dirPrecPct and the declustered TRADED tally were all computed from oPrevSignal - the RAW argmax - while the live order, the arrow and the panel all run on oDeploySignal, which is argmax AFTER the confidence threshold. The gate was certifying a strategy the EA does not trade. Invisible until now: the threshold sat at ~0.02, so the two populations were the same set. The held-out calibration slice (2189316) moved it to 0.14-0.40 and the gap opened immediately - PAI era 256 graded 100% coverage while its traded population was 21% (3,399 of ~16,200 OOS bars). Consequences that were being hidden: - coveragePct >= minCoveragePct was tested against the wrong population, so a model whose TRADED coverage falls under the 24.8% floor still read as clearing it - precSE = sqrt(p(1-p)/n) used n ~16,000 instead of n ~3,400, so the EDGE_MIN_SIGMAS bar was ~2.2x too lenient on the real evidence - the NMS replay declustered a different, larger stream than live, so threshold-rejected bars consumed cluster slots and set alternation state Gate quantities now read m_oosBuyFired/m_oosSellFired (the thresholded population, already tracked for the live-precision line) and the NMS replay runs on oDeploySignal. The threshold can only turn a direction into Neutral, never flip a side, so the fired set is a strict subset and every per-bar outcome is the one already computed. Recall and logBuyPrecPct deliberately stay on the raw argmax: they measure intrinsic class separation, and thresholding them would conflate "cannot separate the classes" with "declines to act on the separation it found". This is the 9a7c37f defect class, and the NMS block carried a comment warning about it while committing it three lines above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 19:34:32 -04:00
//--- oDeploySignal, NOT oPrevSignal: live NMS runs downstream of the confidence
//--- threshold (RefreshLatestSignal feeds NmsLiveAccept the ADJUSTED decision), so
//--- replaying it on the raw argmax declusters a different, strictly larger stream
//--- than the EA ever sees - different survivors, not just more of them, because rule 1
//--- collapses runs and rule 3 alternates over whatever sequence it is given. Bars the
//--- threshold rejects must not consume a cluster slot or set the alternation state.
ENUM_SIGNAL nmsDir = DoubleToSignal(oDeploySignal);
fix: NMS gates the TRADE, not just the arrow - one arrow is now one trade NmsLiveAccept() appeared in exactly one place: wrapped around DrawObject(). It never touched dPrevSignal, and dPrevSignal is what LongCondition() / ShortCondition() / SignedAIConfidence() read. So a declustered bar lost its arrow and still opened a position. Measured on SP500 H1 2026-08-09: CONV called a direction on 64% of bars, so the ~500 bars visible on screen held ~320 decisions - and ~40 arrows were drawn. Roughly one arrow per eight positions the EA would take. And the survivors are not a random eighth. Rule 2 of the declustering keeps the HIGHER-CONFIDENCE side of a cluster, so the visible set is systematically the best member of each run. A chart showing the best of every eight decisions and hiding the rest reads far better than the model is - the same best-of-N selection error already corrected in the geometry scan, the indicator tuner, the lag profile and the deploy gate, this time on the display layer, where it is most likely to mislead the person deciding whether to trade. Fixed by neutralising dPrevSignal when NMS rejects, rather than adding a "may trade" flag consulted at each read site: that leaves exactly ONE definition of what the model decided this bar, so the arrow, the panel's "Current signal", the confidence feeding sizing/SL/TP/trailing, the refresh tally and the order itself cannot drift apart again. Also reports the consequence instead of hiding it. Every OOS counter on the era line still scores every directional call - a population ~8x larger than what now trades - so the line carries a second figure: | TRADED (declustered) NN% on N calls (edge +Npp) replaying the identical rule over pass 3 (which walks OOS bars oldest to newest, the same order the live sweep sees). Its cursors are separate members from the live ones so a training pass can never disturb the live chart's declustering. Deliberately NOT switched into selectionScore yet. Declustering cuts coverage from ~64% of bars to ~8%, well under MIN_COVERAGE_FRACTION_OF_BASE_RATE, which would make every checkpoint undeployable overnight - the minRR collision and the recall-floor catch-22 twice over. The floor gets re-derived from these measurements first. Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-09 10:22:31 -04:00
if(nmsDir == Buy || nmsDir == Sell)
{
fix: the deploy gate graded the un-thresholded model coveragePct, dirPrecPct and the declustered TRADED tally were all computed from oPrevSignal - the RAW argmax - while the live order, the arrow and the panel all run on oDeploySignal, which is argmax AFTER the confidence threshold. The gate was certifying a strategy the EA does not trade. Invisible until now: the threshold sat at ~0.02, so the two populations were the same set. The held-out calibration slice (2189316) moved it to 0.14-0.40 and the gap opened immediately - PAI era 256 graded 100% coverage while its traded population was 21% (3,399 of ~16,200 OOS bars). Consequences that were being hidden: - coveragePct >= minCoveragePct was tested against the wrong population, so a model whose TRADED coverage falls under the 24.8% floor still read as clearing it - precSE = sqrt(p(1-p)/n) used n ~16,000 instead of n ~3,400, so the EDGE_MIN_SIGMAS bar was ~2.2x too lenient on the real evidence - the NMS replay declustered a different, larger stream than live, so threshold-rejected bars consumed cluster slots and set alternation state Gate quantities now read m_oosBuyFired/m_oosSellFired (the thresholded population, already tracked for the live-precision line) and the NMS replay runs on oDeploySignal. The threshold can only turn a direction into Neutral, never flip a side, so the fired set is a strict subset and every per-bar outcome is the one already computed. Recall and logBuyPrecPct deliberately stay on the raw argmax: they measure intrinsic class separation, and thresholding them would conflate "cannot separate the classes" with "declines to act on the separation it found". This is the 9a7c37f defect class, and the NMS block carried a comment warning about it while committing it three lines above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 19:34:32 -04:00
//--- Confidence for rule 2's cross-direction resolution comes from the same adjusted
//--- decision, matching NmsLiveAccept's input exactly.
double nmsConf = MathAbs(oDeploySignal);
fix: NMS gates the TRADE, not just the arrow - one arrow is now one trade NmsLiveAccept() appeared in exactly one place: wrapped around DrawObject(). It never touched dPrevSignal, and dPrevSignal is what LongCondition() / ShortCondition() / SignedAIConfidence() read. So a declustered bar lost its arrow and still opened a position. Measured on SP500 H1 2026-08-09: CONV called a direction on 64% of bars, so the ~500 bars visible on screen held ~320 decisions - and ~40 arrows were drawn. Roughly one arrow per eight positions the EA would take. And the survivors are not a random eighth. Rule 2 of the declustering keeps the HIGHER-CONFIDENCE side of a cluster, so the visible set is systematically the best member of each run. A chart showing the best of every eight decisions and hiding the rest reads far better than the model is - the same best-of-N selection error already corrected in the geometry scan, the indicator tuner, the lag profile and the deploy gate, this time on the display layer, where it is most likely to mislead the person deciding whether to trade. Fixed by neutralising dPrevSignal when NMS rejects, rather than adding a "may trade" flag consulted at each read site: that leaves exactly ONE definition of what the model decided this bar, so the arrow, the panel's "Current signal", the confidence feeding sizing/SL/TP/trailing, the refresh tally and the order itself cannot drift apart again. Also reports the consequence instead of hiding it. Every OOS counter on the era line still scores every directional call - a population ~8x larger than what now trades - so the line carries a second figure: | TRADED (declustered) NN% on N calls (edge +Npp) replaying the identical rule over pass 3 (which walks OOS bars oldest to newest, the same order the live sweep sees). Its cursors are separate members from the live ones so a training pass can never disturb the live chart's declustering. Deliberately NOT switched into selectionScore yet. Declustering cuts coverage from ~64% of bars to ~8%, well under MIN_COVERAGE_FRACTION_OF_BASE_RATE, which would make every checkpoint undeployable overnight - the minRR collision and the recall-floor catch-22 twice over. The floor gets re-derived from these measurements first. Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-09 10:22:31 -04:00
int lastSame = (nmsDir == Buy) ? m_oosNmsLastBuyIdx : m_oosNmsLastSellIdx;
//--- 1) same-direction contiguous collapse; last-seen advances either way so a whole
//--- run collapses to its first bar.
bool cont = (lastSame >= 0 && (lastSame - oi) <= m_signalClusterWindow);
if(nmsDir == Buy)
m_oosNmsLastBuyIdx = oi;
else
m_oosNmsLastSellIdx = oi;
bool keep = !cont;
//--- 2) cross-direction resolution against the last KEPT opposite signal: flicker at
//--- one turn zone resolves to the more confident side.
if(keep && m_oosNmsKeptIdx >= 0 && m_oosNmsKeptDir != nmsDir &&
m_oosNmsKeptDir != Neutral && (m_oosNmsKeptIdx - oi) <= m_signalClusterWindow)
keep = (nmsConf > m_oosNmsKeptConf);
feat: 10-bar decluster window + alternation on every signal consumer SignalClusterWindow 3 -> 10 for all topologies. On H1 a 3-bar window collapsed only the tightest runs and left visible clusters at every turn; 10 bars is closer to the spacing of genuinely distinct setups. ALTERNATION. Rule 1 only collapses a same-direction run INSIDE the window; past it a second Buy is emitted with no Sell between, giving Buy/Buy/Buy/Sell. With both directions tradeable that sequence is the model re-entering a move it is already in rather than finding a new one. The kept sequence must now alternate: the first signal passes, and after that a direction passes only if the last KEPT signal was the opposite one. Added to ALL THREE consumers, with identical logic, because they must agree: - NmsLiveAccept -> the live trade - pass 3's OOS replay -> the tally the deploy gate grades - PruneDirectionalClusters -> the drawn history A rule applied to only some of these certifies one strategy and trades another - the same defect class as the geometry the gate certified while OpenParams placed something else (9a7c37f) - and would draw the user arrows the EA would never have taken. Deliberately NOT applied to the LABEL. The barrier target has no "must flip" invariant: consecutive Buy labels are routinely correct, and an earlier alternation gate was removed with the triple-barrier relabel for exactly that reason. This filters what is ACTED ON, which is what "applies to training" can honestly mean here - pass 3's declustered tally is the training-side number that decides deployment. BothDirectionsTradeable() is the stated precondition (with one side disabled there is no opposite to wait for, so alternation would suppress everything after the first call). This build has no long-only/short-only input, so it is constant true - kept as a named predicate so a future direction restriction has one place to change rather than three call sites silently assuming both sides. Build tag -> nms-alternate-v4. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 14:26:12 -04:00
//--- 3) ALTERNATION, identical to NmsLiveAccept's rule 3. MUST match it exactly:
//--- this tally is what the deploy gate grades, so any divergence certifies one
//--- strategy and trades another - the same class of defect as the geometry the
//--- gate certified while OpenParams placed something else (9a7c37f).
if(keep && BothDirectionsTradeable() && m_oosNmsKeptIdx >= 0 &&
m_oosNmsKeptDir == nmsDir)
keep = false;
fix: NMS gates the TRADE, not just the arrow - one arrow is now one trade NmsLiveAccept() appeared in exactly one place: wrapped around DrawObject(). It never touched dPrevSignal, and dPrevSignal is what LongCondition() / ShortCondition() / SignedAIConfidence() read. So a declustered bar lost its arrow and still opened a position. Measured on SP500 H1 2026-08-09: CONV called a direction on 64% of bars, so the ~500 bars visible on screen held ~320 decisions - and ~40 arrows were drawn. Roughly one arrow per eight positions the EA would take. And the survivors are not a random eighth. Rule 2 of the declustering keeps the HIGHER-CONFIDENCE side of a cluster, so the visible set is systematically the best member of each run. A chart showing the best of every eight decisions and hiding the rest reads far better than the model is - the same best-of-N selection error already corrected in the geometry scan, the indicator tuner, the lag profile and the deploy gate, this time on the display layer, where it is most likely to mislead the person deciding whether to trade. Fixed by neutralising dPrevSignal when NMS rejects, rather than adding a "may trade" flag consulted at each read site: that leaves exactly ONE definition of what the model decided this bar, so the arrow, the panel's "Current signal", the confidence feeding sizing/SL/TP/trailing, the refresh tally and the order itself cannot drift apart again. Also reports the consequence instead of hiding it. Every OOS counter on the era line still scores every directional call - a population ~8x larger than what now trades - so the line carries a second figure: | TRADED (declustered) NN% on N calls (edge +Npp) replaying the identical rule over pass 3 (which walks OOS bars oldest to newest, the same order the live sweep sees). Its cursors are separate members from the live ones so a training pass can never disturb the live chart's declustering. Deliberately NOT switched into selectionScore yet. Declustering cuts coverage from ~64% of bars to ~8%, well under MIN_COVERAGE_FRACTION_OF_BASE_RATE, which would make every checkpoint undeployable overnight - the minRR collision and the recall-floor catch-22 twice over. The floor gets re-derived from these measurements first. Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-09 10:22:31 -04:00
if(keep)
{
m_oosNmsKeptIdx = oi;
m_oosNmsKeptDir = nmsDir;
m_oosNmsKeptConf = nmsConf;
m_oosNmsFired++;
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
//--- oTradeWon, not `hit`: this pair exists specifically to answer "what would I
//--- have made", and that is a question about the trade, not about the label.
fix: the deploy gate graded the un-thresholded model coveragePct, dirPrecPct and the declustered TRADED tally were all computed from oPrevSignal - the RAW argmax - while the live order, the arrow and the panel all run on oDeploySignal, which is argmax AFTER the confidence threshold. The gate was certifying a strategy the EA does not trade. Invisible until now: the threshold sat at ~0.02, so the two populations were the same set. The held-out calibration slice (2189316) moved it to 0.14-0.40 and the gap opened immediately - PAI era 256 graded 100% coverage while its traded population was 21% (3,399 of ~16,200 OOS bars). Consequences that were being hidden: - coveragePct >= minCoveragePct was tested against the wrong population, so a model whose TRADED coverage falls under the 24.8% floor still read as clearing it - precSE = sqrt(p(1-p)/n) used n ~16,000 instead of n ~3,400, so the EDGE_MIN_SIGMAS bar was ~2.2x too lenient on the real evidence - the NMS replay declustered a different, larger stream than live, so threshold-rejected bars consumed cluster slots and set alternation state Gate quantities now read m_oosBuyFired/m_oosSellFired (the thresholded population, already tracked for the live-precision line) and the NMS replay runs on oDeploySignal. The threshold can only turn a direction into Neutral, never flip a side, so the fired set is a strict subset and every per-bar outcome is the one already computed. Recall and logBuyPrecPct deliberately stay on the raw argmax: they measure intrinsic class separation, and thresholding them would conflate "cannot separate the classes" with "declines to act on the separation it found". This is the 9a7c37f defect class, and the NMS block carried a comment warning about it while committing it three lines above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 19:34:32 -04:00
//--- Safe to reuse even though oTradeWon is keyed to oPrevSignal's direction: the
//--- threshold only ever turns a direction into Neutral, so reaching here at all
//--- means oDeploySignal and oPrevSignal name the SAME side.
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
if(oTradeWon)
fix: NMS gates the TRADE, not just the arrow - one arrow is now one trade NmsLiveAccept() appeared in exactly one place: wrapped around DrawObject(). It never touched dPrevSignal, and dPrevSignal is what LongCondition() / ShortCondition() / SignedAIConfidence() read. So a declustered bar lost its arrow and still opened a position. Measured on SP500 H1 2026-08-09: CONV called a direction on 64% of bars, so the ~500 bars visible on screen held ~320 decisions - and ~40 arrows were drawn. Roughly one arrow per eight positions the EA would take. And the survivors are not a random eighth. Rule 2 of the declustering keeps the HIGHER-CONFIDENCE side of a cluster, so the visible set is systematically the best member of each run. A chart showing the best of every eight decisions and hiding the rest reads far better than the model is - the same best-of-N selection error already corrected in the geometry scan, the indicator tuner, the lag profile and the deploy gate, this time on the display layer, where it is most likely to mislead the person deciding whether to trade. Fixed by neutralising dPrevSignal when NMS rejects, rather than adding a "may trade" flag consulted at each read site: that leaves exactly ONE definition of what the model decided this bar, so the arrow, the panel's "Current signal", the confidence feeding sizing/SL/TP/trailing, the refresh tally and the order itself cannot drift apart again. Also reports the consequence instead of hiding it. Every OOS counter on the era line still scores every directional call - a population ~8x larger than what now trades - so the line carries a second figure: | TRADED (declustered) NN% on N calls (edge +Npp) replaying the identical rule over pass 3 (which walks OOS bars oldest to newest, the same order the live sweep sees). Its cursors are separate members from the live ones so a training pass can never disturb the live chart's declustering. Deliberately NOT switched into selectionScore yet. Declustering cuts coverage from ~64% of bars to ~8%, well under MIN_COVERAGE_FRACTION_OF_BASE_RATE, which would make every checkpoint undeployable overnight - the minRR collision and the recall-floor catch-22 twice over. The floor gets re-derived from these measurements first. Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-09 10:22:31 -04:00
m_oosNmsHits++;
}
}
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// Same confusion counts keyed by what the model actually PREDICTED this bar, not the
// true label - see m_oosBuyPredicted's declaration comment for why recall alone can
// hide an over-firing class.
switch(DoubleToSignal(oPrevSignal))
{
case Buy:
m_oosBuyPredicted++;
if(hit)
m_oosBuyPredictedHits++;
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
if(oWinLong)
m_oosBuyPredictedWins++;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
break;
case Sell:
m_oosSellPredicted++;
if(hit)
m_oosSellPredictedHits++;
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
if(oWinShort)
m_oosSellPredictedWins++;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
break;
default:
m_oosNeutralPredicted++;
if(hit)
m_oosNeutralPredictedHits++;
break;
}
// Live-decision precision: scores the bars on which the deployed EA would actually cast a
// directional vote, using the prior-corrected (logit-adjusted) posterior - see
// AdjustedSignalFromSoftmax()/RefreshLatestSignal(). The recall/argmax-precision above stay
// on the raw argmax (the model's intrinsic class separation, which the convergence gate
// needs); THIS scores what trades live, so the panel's live precision number is the
// precision a buyer gets forward. TempData still holds this bar's raw softmax probs
// (nothing overwrote them since ApplyClassificationSoftmax above), so the adjustment reads
// them directly. Neutral picks aren't counted - they cast no vote.
// No confidence-floor term any more: with the floor removed, EVERY non-Neutral adjusted
// decision casts a vote (at its tier weight), so any threshold here would score a
// different population than the one that actually votes. Whether a given vote goes on to
// OPEN a position additionally depends on Min_Vote_Open versus the AVERAGE across all
// voting filters, which this per-bar training-time scorer has no visibility of - so this
// stays the honest "would have voted, and was it right" measure rather than pretending to
// model the aggregate.
if(m_outputNeuronsCount == 3)
{
double adjSig = AdjustedSignalFromSoftmax();
ENUM_SIGNAL adjEnum = DoubleToSignal(adjSig);
if(adjEnum != Neutral)
{
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
//--- Same substitution as everywhere else in this block: what a buyer gets forward is
//--- whether the trade paid, not whether it agreed with a collapsed label.
bool fireHit = (adjEnum == Buy) ? oWinLong : oWinShort;
feat(rank): AI models rank their own confidence tiers from held-out outcomes Closes the caveat 4858507 shipped with: the vote is a confidence percentage, but only to the extent the pattern weights are measured. AI tier weights sat at their designed defaults (25/50/75/100) because AI rows only ever arrive from LIVE journaling, of which a training run produces almost none. AND A STALE-TIER BUG THAT MADE THE EVIDENCE MEANINGLESS. The OOS scan bucketed every scanned bar by ConfidenceTier(), which reads dPrevSignal - and dPrevSignal is assigned in PASS 1 only, never anywhere in the OOS scan. So an entire era's fires were bucketed by one stale, unrelated bar's confidence and landed in a SINGLE tier. That is the "tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0)" symptom recorded on 2026-08-16 and attributed to the calibration clamp. The clamp was real and was fixed then; this is a second, independent cause of the identical output that survived that fix untouched - which is why the log kept reading the same afterwards. Two causes, one symptom. Now ConfidenceTierFor(adjSig): the bar this iteration actually scored. WHY THIS DOES NOT WRITE ROWS TO THE SIGNAL DB, which was the obvious reading of "fill the database during training". The user's own observation is the reason: a classic Pattern_2 is a fixed geometric condition, so its win rate is legitimately accumulated over years, but an AI Pattern_2 means "confidence landed in tier 2" and tier 2 under era 100's weights is a different statement from tier 2 under era 500's. The DB's value is ACCUMULATION, and accumulation is exactly what is wrong here - it would average together models that no longer exist, while colliding with the per-table row cap and mixing measured-on-holdout outcomes into the live ledger's own tables. What the DB actually supplies is a measured win rate per pattern, and pass 3 already computes that on held-out bars, thousands at a time. So the model ranks itself once per era, REPLACING rather than accumulating, which makes the weights describe the current weights by construction. ESTIMATOR. Not WinRateFromCounts(): it returns NO_DATA below 100 raw trades BEFORE shrinking, which here would fire on every tier every era and hand all four the pooled rate - the tiers could never separate and the mechanism would be inert. Shrinkage is the answer to a small sample; a floor in front of it means the shrinkage never runs. Instead: a Beta prior of TIER_PRIOR_EFF_N pseudo-observations centred on the model's pooled holdout rate, counted in EFFECTIVE observations, because overlapping triple-barrier labels mean 800 raw fires can be worth ~12 independent ones. Rounded to the integer, not to the decade NormalizeWinRate() uses, which would collapse the shrunk tiers back into one number. NO SAME-ERA CIRCULARITY, and it falls out of the ordering rather than a guard: weights are computed at the END of era N, so the vote scored during era N was cast with era N-1's weights. The deploy gate never grades a vote whose weights were fitted on the bars it is scoring. Residual leakage remains - the same OOS bars each era under a different model - and is stated in the code rather than papered over. Both DB clobber paths are closed: ApplyPatternWeight() declines once self-ranked, and UpdateSignalsWeights()' filter.Weight() call is guarded by SelfRanked() - guarding only the tiers would have let the hourly ranking pass undo half the self-ranking. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 16:00:32 -04:00
//--- Bucket the same fire by confidence tier - see m_oosTierFired. THE ARGUMENT
//--- MATTERS: this was ConfidenceTier(), whose comment claimed it "reads the net's
//--- CURRENT outputs, which is exactly the bar AdjustedSignalFromSoftmax() just
//--- scored". It does not. ConfidenceTier() reads dPrevSignal, and dPrevSignal is
//--- assigned in PASS 1 only (the in-sample pass) - never anywhere in this OOS
//--- scan. So every scanned bar of the era was bucketed by one stale, unrelated
//--- bar's confidence, and the whole era's fires landed in a SINGLE tier.
//---
//--- That is the "tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0)" symptom
//--- recorded on 2026-08-16 and attributed to the calibration clamp. The clamp was
//--- a real cause and was fixed then (raw magnitude, see ConfidenceTierFor); this
//--- is a SECOND, independent cause that produces the identical single-bucket
//--- output and survived that fix untouched. Two causes, one symptom - which is
//--- why the log kept reading the same after the first was closed.
//---
//--- adjSig is the bar this iteration actually scored, and fireHit below is derived
//--- from it, so tier and outcome now describe the same bar by construction.
int fireTier = ConfidenceTierFor(adjSig);
feat(ai): measure precision per confidence tier; fix stale metric labels Two things the 2026-07-30 run exposed. 1. Every user-facing message still called the selection metric "balanced accuracy". It has ranked on directional precision since a142749, so "CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5% PRECISION as if it were macro-recall, while the same era logged an actual balanced accuracy of 49%. Two different numbers under one name, in the line that announces a deploy. Relabelled at every site, including the stage-3 refusal, which still described the per-class recall floor that stopped being the gate. 2. Precision is now bucketed by confidence tier and logged per era, both per-tier and cumulatively from each tier upward: | tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ... The per-tier number says whether confidence is calibrated to correctness at all; if it does not rise T0->T3, raising the floor buys nothing and that is the finding. The ">=" number is what a floor would actually deliver, with its fire count, so the coverage cost is visible in the same line. Tier weights are 25/50/75/100, so for an AI-only config Min_Vote_Open maps straight across: 50 = ">=T1", 75 = ">=T2", 100 = ">=T3". Bucketing happens at the existing live-fired accounting site, so it measures exactly the population that trades - not the raw argmax. Both builds compile 0 errors, 0 warnings. No retrain needed for either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
if(fireTier >= 0 && fireTier < 4)
{
m_oosTierFired[fireTier]++;
if(fireHit)
m_oosTierHits[fireTier]++;
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
if(adjEnum == Buy)
{
m_oosBuyFired++;
if(fireHit)
m_oosBuyFiredHits++;
}
else
{
m_oosSellFired++;
if(fireHit)
m_oosSellFiredHits++;
}
}
}
if(hit)
{
dOosForecast += (100 - dOosForecast) / Net.recentAverageSmoothingFactor;
dOosError -= dOosError / Net.recentAverageSmoothingFactor;
}
else
{
dOosForecast -= dOosForecast / Net.recentAverageSmoothingFactor;
dOosError += (100 - dOosError) / Net.recentAverageSmoothingFactor;
}
}
perf: pass 1 forward-passed ~40% of bars that a later pass redid anyway Pass 1 already skipped its feedForward on QUEUED bars, because pass 2 redoes them. The same argument covers two more bands it was still forwarding: OOS window (30% of bars) - pass 3 re-forwards every one of them calibration band (~10% of bars) - pass 2.5 re-forwards every one of them All three passes derive their bounds from the same helpers and apply the identical eligibility test, so the bar sets are equal by construction, not by coincidence. Only the two purge bands and the ineligible edge bars are visited in pass 1 and nowhere else - those keep their forward pass. The scan's copy was never the one that survived. Its arrow-cache write was overwritten by pass 3's (with the thresholded, post-training decision), its status-label paint was transient, and its predicted-class tally measured last era's weights. Those tallies move to pass 2.5 and pass 3, on the raw argmax exactly as pass 1 and pass 2 count it, so the population behind the panel's "Predicted -> Buy/Sell/Neutral" line is unchanged and stays comparable with the "Actual" line beside it, which pass 1 still accumulates over every labelled bar. Verified unaffected by the cut: dPrevSignal and m_lastBarTime are both written last by bars 0/1, which are label-ineligible and therefore still forwarded, so FinalizeTrainRun's `dtStudied = m_lastBarTime` and Lifecycle's newBarPending sentinel read the same values as before. Correctness, not just speed: batch norm is UNFROZEN during pass 1 (passes 2.5 and 3 freeze it deliberately), so every scan-time forward on a held-out bar was advancing the BN running mean/variance from data the model is graded on. Those running statistics are inference-time model state. It is the mild, unsupervised kind of leakage - feature statistics, not labels - but it fed the weights pass 3 then scored, and it is now gone. Cost: ~40% of all bars lose one forward pass per era, ~16% of net time once pass 2's backward pass is weighted in. Per-dispatch, so it lands on every backend. Both variants compile 0 errors / 0 warnings. Build tag scan-nofwd-v5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:50:14 -04:00
//--- Predicted-class tally for the OOS window, which pass 1 used to compute from its own
//--- (now-removed) redundant feedForward on this same bar - see the laterPassForwards
//--- comment there. On the RAW argmax, exactly as pass 1 and pass 2 count it: this pair of
//--- panel lines reports what the model called versus what was true, so it must not be
//--- silently narrowed to the thresholded decision on one third of the bars.
switch(DoubleToSignal(oPrevSignal))
{
case Buy:
m_countBuySignals++;
break;
case Sell:
m_countSellSignals++;
break;
default:
m_countNeutralSignals++;
break;
}
// Chart annotation for this (OOS) bar, using post-training weights - pass 1 no longer
// draws these at all (it used to, from a pre-training snapshot that this then overwrote).
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_lastBarTime = m_Time.GetData(oi);
if(oi > 0)
{
// NMS on: record only (the era-end sweep renders); off: draw inline. See pass 1's note.
if(m_signalClusterWindow > 0)
{
if(oi < ArraySize(m_arrowSignalCache))
m_arrowSignalCache[oi] = oDeploySignal;
}
else
if(DoubleToSignal(oDeploySignal) == Neutral)
DeleteObject(m_lastBarTime);
else
feat(chart): signal marks become price LEVELS at the trigger, not arrows beside the candle User request: 'move from arrows on lows and highs to small horizontal lines at the actual prices the entry/exit would trigger, just a bit larger than the candles. dark green for buy, dark red for sell.' Every mark is now an OBJ_TREND segment with both anchors at one price and both rays off, spanning 1.3 bar widths, drawn at the bar's CLOSE - the price a market order actually fires at, and the exact entry TripleBarrierLabel assumes. It used to sit on the candle's LOW for a Buy and its HIGH for a Sell: prices the trade never touches, picked so an arrow glyph would clear the candle. The tooltip now carries that price too. COLOUR NOW MEANS DIRECTION AND ONLY DIRECTION on every layer (dark green / dark red). Layer moves to width+style - the traded vote is solid and thick and drawn in front, a single model's raw opinion is thin, dotted and behind the candles - which keeps the distinction the old palette existed to draw (a model's opinion must never read as a trade) while freeing colour to say one thing consistently. Consequences handled, all of them the same 'a typed scan went blind' failure: - SaveChartSignals filtered OBJPROP_TYPE == OBJ_ARROW and read OBJPROP_ARROWCODE. It now filters OBJ_TREND and recovers direction from the colour. The sidecar keeps the old 217/218 numbers as its buy/sell token deliberately, so existing .arrows files still load. - AdvanceChartSignalRestore now rebuilds through the SAME creation point the live path uses, so a restored mark and a fresh one are identical objects. - The rescan-scoped delete enumerated ObjectsTotal(OBJ_ARROW) - retyped, or it silently deletes nothing. - ApplySignalsVisibility enumerated OBJ_ARROW with NO prefix filter. Under the new type that would have hidden and shown THE USER'S OWN trend lines on every Hide/Show click; it is now prefix-scoped. The old type was uncommon enough on a real chart to mask the missing check - trend lines are the most hand-drawn object there is. - DrawObject's high/low parameters are gone (6 call sites pass m_Close instead), so no caller can hand it a price it no longer draws at. - Fixed a pre-existing stale comment that still described the purge sweep as OBJ_ARROW-only three lines above the note explaining it had been widened to every type. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 14:11:49 -04:00
DrawObject(m_lastBarTime, oDeploySignal, m_Close.GetData(oi));
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
}
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
} // end direction (non-meta) OOS scoring body
fix(deinit): vote arrows survived the cheap sweep, and 5 long loops ignored the stop Leftover chart objects on long-history charts. Two causes, one of them introduced by 07aa017. THE ONE I ADDED. The filtered view's overlay draws up to SIGNAL_RESCAN_LOOKBACK_BARS vote arrows. OnDeinit's EARLY VISIBLE-UI SWEEP runs with skipArrows=true, which skips any prefix equal to SIG_ARROW_PREFIX - and "WarSig_VOTE_..." starts with "WarSig_", so every one of them was skipped by the one sweep that is cheap enough to always complete. They then sat in the object list while the two expensive scans that follow walked it: a per-member SaveChartSignals O(total) scan, then the by-name rescan. On a chart with years of history that is thousands of extra objects walked twice, inside a teardown budget measured from the stop REQUEST rather than from OnDeinit's first line. skipArrows exists because the per-model arrows' sidecar is rebuilt by SCANNING them off the chart, so they cannot be deleted before that write. Vote arrows have no sidecar - they are a reconstruction, rebuilt on the next attach - so nothing is preserving them and they now get their own prefix slot, deleted by one native call in the first few milliseconds. THE FIVE LOOPS. A time budget bounds THROUGHPUT, not latency to an unload, and OnDeinit cannot begin until whatever is in flight returns. These all scaled with history and none of them checked: * Training passes 2, 2.5 and 3 yielded only on TRAIN_TIME_BUDGET_MS. Pass 1 has checked IsStopped() all along; the other three never have, and they are the ones that grow with the bar count. Free to fix - the resume state is written either way, so a stopped chunk simply is not re-entered. * PruneDirectionalClusters: the one UNCHUNKED sweep left, once per era over every bar, with its own header noting that raising the training budget cannot help its cost. Now bails outright. * AdvanceChartSignalRestore / AdvanceChartSignalRescan: chunked, but the rescan runs a full feedForward per bar over up to 5000 bars and the restore can hold MAX_RESTORED_ARROWS entries. Checked on the same 64-object stride as the clock read, since the check is not free either. * AdvanceFilteredOverlay (mine, 07aa017) replays Direction() on every classic filter per bar and had no check at all. Now per bar. ChartUI.mqh had ZERO shutdown checks across six loops before this. Nothing was added to the purge path itself: that is the work that must complete, and an IsStopped() check inside it would abort unconditionally - IsStopped() is already true by the time OnDeinit runs. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 17:13:50 -04:00
//--- Time OR stop - see pass 2's matching comment.
if(m_oosScoreIndex - 1 >= 2 && (IsStopped() || GetTickCount() - chunkStartTick >= TRAIN_TIME_BUDGET_MS))
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
//--- yield: save enough to resume PASS 3 mid-walk on the next call - m_isPass3Active and
//--- m_oosScoreIndex (both members) carry the actual resume position.
m_resumeBars = bars;
m_resumeTotalIter = totalIter;
m_resumeOosCutoff = oosCutoff;
m_resumeAddLoop = add_loop;
m_resumeBarIndex = i;
m_eraResumePending = true;
m_modelEta = eta;
return;
}
}
m_isPass3Active = false;
feat(gate): grade OOS calls on the exit policy actually in force, and move vote combining out of the members and into the orchestrator Option (a) from the exit-policy question: the certified number must be the traded number. Plus the modularity correction the user called for on 778b6c0. 1. VOTE COMBINING BELONGS TO THE ORCHESTRATOR, NOT TO A MEMBER. 778b6c0 fixed the last-writer-wins bug on g_LiveAISignedConfidence by having a member average its siblings through g_warriorEnsemble. That trades a scheduling bug for a coupling bug, and it is the wrong shape for this EA: every signal runs in its own instance, minds its own state, and VOTES to the orchestrator, which is the only thing allowed to combine opinions. Replaced with a publish/aggregate pair in Variables\ConfidenceBridge.mqh whose split is enforced by shape rather than by convention: - PublishAIVote(slot, conf) - a member writes ONLY its own slot, reads nobody's; - AggregateAIVotes() - called by CExpertSignalCustom::LiveSignedConfidence. CExpertSignalAIBase::EnsembleLiveSignedConfidence is gone. The orchestrator also republishes the aggregate into g_LiveAISignedConfidence, because the intelligent trailing reads that global directly and must act on the same number the exit route does rather than on a leftover from whichever member ticked last. A solo AI signal owns slot 0, so the non-ensemble path is unchanged. 2. THE GATE NOW REPLAYS THE REAL EXIT RULE. SimulateTradeOutcome() walks the same price series with the same fill/barrier/spread convention as ComputeLabelForBar - deliberately by copy, so a disagreement between the two can only be a policy effect and never a discrepancy between two pieces of our own arithmetic - and terminates at the FIRST of stop / target / vote reversal / horizon. Barriers are tested before the vote on the same bar: intrabar we cannot know which came first, and the barrier is what the broker executes automatically, so checking the vote first would credit the exit policy with escapes a real stop would have taken out of its hands. It runs AFTER pass 3, not inside it. A vote-flip exit for a trade entered at bar r is decided by the model's output at bars r-1, r-2, ... - NEWER bars - and pass 3 walks oldest-to-newest, so at the moment r is graded its own exit does not exist yet. Only once m_oosDecisionSeries is complete over the whole OOS window can the replay run. In ensemble mode that series carries the member's adjusted decision and the live exit reads the ensemble aggregate, which is the coupling the user identified: an LSTM entry really can be closed by the ensemble turning against it. 3. THE STATISTIC HAS TO CHANGE WITH THE POLICY, AND THAT IS THE REAL FINDING. A barrier exit pays a fixed R. A vote exit pays whatever the close happens to be. So the moment vote exits are enabled the payoff is CONTINUOUS, and "win rate vs break-even" stops being a meaningful test - there is no fixed break-even for a variable payoff. SimulateTradeOutcome therefore returns R rather than a bool, and the replay reports expectancy in R with its SE taken from the R distribution (overlap- deflated on the same EffectiveSampleSize doctrine as every other SE here), not from a binomial. This is the same class of error as win-based scoring in 2026-08-09: measuring a variable-payoff process with a fixed-payoff statistic. Naming it now, while vote exits are still off, is much cheaper than discovering it after they go on. 4. WHY THIS IS SAFE TO SHIP TODAY. Min_Vote_Close ships Disabled (101, unreachable on both scales it drives) and reaches the AI signal through the new ExitPolicy() setter as 1.01, which the setter turns into 0.0 = no vote exit. Under that policy every replayed trade resolves at a barrier and the simulation is arithmetically the same trade the deploy gate already certifies - they cannot drift. The report says so explicitly, and prints ONCE per run in that state; when vote exits are on it prints every era, because then the divergence is the thing to watch. Nothing about today's numbers moves. The gate switchover is wired but dormant by construction: it becomes exit-aware the moment the input is enabled, which is exactly what "the certified number is the traded number" has to mean. KNOWN LIMIT, stated rather than hidden: only the AI early-exit route is replayed. The rule-based averaged-vote close (m_threshold_close) depends on every other filter's live vote, which pass 3 does not reproduce, so a position the classic filters would have closed is held to its barrier here. The replay therefore holds LONGER than live and overstates barrier-reached outcomes. Faithful only while the AI is the dominant vote - which is the configuration this is being built for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:25:57 -04:00
//--- THE EXIT SIMULATION, and it has to run HERE rather than inline in the scan above. A vote-flip
//--- exit for a trade entered at bar r is decided by what the model says at bars r-1, r-2, ... -
//--- which are NEWER bars, and pass 3 walks oldest-to-newest (m_oosScoreIndex descends, a high
//--- index is an old bar). So at the moment bar r is graded its own exit has not been decided yet.
//--- Only now is m_oosDecisionSeries complete over the whole OOS window.
SimulateExitPolicyOutcomes();
ReportExitPolicyDivergence();
feat: excursion-size head (Stage 1, measurement only) Direction is closed - normalised asymmetry fails on three instruments with a working positive control, and the classifier's own best-of-999 era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is a different question and RANGE clears at ~4x its null. Checked the denomination before building on that, since the source memo warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is predictable" is a claim about travel RELATIVE to current ATR, not a restatement of "ATR is autocorrelated". It is exactly the part a fixed multiple (stop 3.31*ATR, target 1.64*ATR) discards. A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches ladder rung k) upward and downward. Survival parameterisation rather than regressing the multiple, because it needs nothing new from CNet: sigmoid outputs and the per-neuron delta the `total != 3` branch already applies (a quantile head would need a linear activation and a pinball gradient in Network.mqh, Network.cl and the DirectML path, on a class four topologies share). Targets are free - m_ladderUpAt already records first-touch age per rung with 0 meaning never reached. Separate net, not extra outputs on the classifier: more outputs would change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push the count off 3 - the exact condition backProp uses to select the joint softmax gradient the 3-class head depends on. The classifier is bit-for-bit unaffected and this is removable without trace. STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the constant per-rung base rate - the baseline a fixed ATR multiple already assumes - with both predictors fitted IS and evaluated OOS, so neither gets a look at the test set. Positive skill justifies Stage 2 (drive SL/TP and sizing off ExcursionQuantile, which is defined and deliberately uncalled). Zero or negative means ATR already carries everything and Stage 2 must not be built. Trains only on primary occurrences: the replay queue oversamples for CLASS balance, and a direction-balanced sample is a biased SIZE sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
//--- Excursion head's verdict for this era, printed while its accumulators are complete and
//--- before the next era's fresh-era block clears them.
ExcursionReport();
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//--- Scoring finished - resume the normal always-adapting statistics (see the freeze at pass-3
//--- start) before anything else runs a forward pass.
Net.SetBatchNormFrozen(false);
revert(ui): restore the unconditional era-end arrow repaint b197999 gated the repaint on "this era beat the best checkpoint". That was not what was asked for and it changes policy rather than fixing a bug, so it is reverted - behaviour is now byte-identical to before b197999. Only a comment recording what was verified remains. What the check found: there is NO repaint or erase at the start of an era. With NMS on, passes 1, 2.5 and 3 only RECORD predictions into m_arrowSignalCache - pass 1's own comment says "NMS on: record only - the era-end sweep is the SOLE renderer, so no raw (un-declustered) arrow is ever drawn mid-era" - and PruneDirectionalClusters runs once, at the end of pass 3, which is the end of the era. The requested behaviour was already the implemented behaviour. So the arrows vanishing at the era boundary is not a timing fault. That sweep DELETEs the arrow on any bar the era scored Neutral, and the model is currently scoring Neutral on 98-100% of bars (see the logit-adjustment finding: Neutral is the RAREST class at 10.6% and the imbalance correction is subsidising it by ~1.2 logits). The chart is reporting the model accurately; the model is the problem. One thing that CAN clear arrows at era 0, and did on the first attach after the |ALTW re-key: ClearPersistedChartSignals("fresh topology at era 0 - arrows belong to a previous model"). That fires once per fresh model, not per era. Compile-verified: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 22:41:59 -04:00
//--- Pass 3 done => every scored bar's prediction is now in m_arrowSignalCache. Collapse each
//--- same-direction cluster to its earliest bar so the chart shows one arrow per real turn.
//--- ONCE PER ERA, AT ITS END, and this is the only place the historical arrows are rendered:
//--- with NMS on, passes 1/2.5/3 only RECORD into the cache (see the "NMS on: record only"
//--- comment in pass 1), and nothing repaints or erases at the start of an era. Verified
//--- 2026-08-16 after a report of arrows vanishing when the next era begins - the vanishing is
//--- this sweep deleting arrows on bars the era genuinely scored Neutral, not a timing fault.
PruneDirectionalClusters(bars);
feat(rank): AI models rank their own confidence tiers from held-out outcomes Closes the caveat 4858507 shipped with: the vote is a confidence percentage, but only to the extent the pattern weights are measured. AI tier weights sat at their designed defaults (25/50/75/100) because AI rows only ever arrive from LIVE journaling, of which a training run produces almost none. AND A STALE-TIER BUG THAT MADE THE EVIDENCE MEANINGLESS. The OOS scan bucketed every scanned bar by ConfidenceTier(), which reads dPrevSignal - and dPrevSignal is assigned in PASS 1 only, never anywhere in the OOS scan. So an entire era's fires were bucketed by one stale, unrelated bar's confidence and landed in a SINGLE tier. That is the "tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0)" symptom recorded on 2026-08-16 and attributed to the calibration clamp. The clamp was real and was fixed then; this is a second, independent cause of the identical output that survived that fix untouched - which is why the log kept reading the same afterwards. Two causes, one symptom. Now ConfidenceTierFor(adjSig): the bar this iteration actually scored. WHY THIS DOES NOT WRITE ROWS TO THE SIGNAL DB, which was the obvious reading of "fill the database during training". The user's own observation is the reason: a classic Pattern_2 is a fixed geometric condition, so its win rate is legitimately accumulated over years, but an AI Pattern_2 means "confidence landed in tier 2" and tier 2 under era 100's weights is a different statement from tier 2 under era 500's. The DB's value is ACCUMULATION, and accumulation is exactly what is wrong here - it would average together models that no longer exist, while colliding with the per-table row cap and mixing measured-on-holdout outcomes into the live ledger's own tables. What the DB actually supplies is a measured win rate per pattern, and pass 3 already computes that on held-out bars, thousands at a time. So the model ranks itself once per era, REPLACING rather than accumulating, which makes the weights describe the current weights by construction. ESTIMATOR. Not WinRateFromCounts(): it returns NO_DATA below 100 raw trades BEFORE shrinking, which here would fire on every tier every era and hand all four the pooled rate - the tiers could never separate and the mechanism would be inert. Shrinkage is the answer to a small sample; a floor in front of it means the shrinkage never runs. Instead: a Beta prior of TIER_PRIOR_EFF_N pseudo-observations centred on the model's pooled holdout rate, counted in EFFECTIVE observations, because overlapping triple-barrier labels mean 800 raw fires can be worth ~12 independent ones. Rounded to the integer, not to the decade NormalizeWinRate() uses, which would collapse the shrunk tiers back into one number. NO SAME-ERA CIRCULARITY, and it falls out of the ordering rather than a guard: weights are computed at the END of era N, so the vote scored during era N was cast with era N-1's weights. The deploy gate never grades a vote whose weights were fitted on the bars it is scoring. Residual leakage remains - the same OOS bars each era under a different model - and is stated in the code rather than papered over. Both DB clobber paths are closed: ApplyPatternWeight() declines once self-ranked, and UpdateSignalsWeights()' filter.Weight() call is guarded by SelfRanked() - guarding only the tiers would have let the hourly ranking pass undo half the self-ranking. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 16:00:32 -04:00
//--- ...and now that this era's per-tier outcomes are complete, turn them into the vote weights
//--- the NEXT era (and live trading) will use. See RankTiersFromOos().
RankTiersFromOos();
feat(consistency): the five review flaws fixed - training wears the live constraints, the gate wears the policy 1. Labels and the exit simulator go through the broker's stop-distance check: risk/reward widen to SYMBOL_TRADE_STOPS_LEVEL exactly as TCAdjustStops does at order time - the M5/tight-ATR case where live trades ran wider geometry than training measured. Current stops level stands in for history (like the spread); measured quantity, so it does not key the fingerprint. 2. The Intelligent drift verdict moved into RefreshDriftVerdict(), which RESCANS the label cache and now runs at every era end beside RankTiersFromOos - era-cadence instead of waiting for rare full rebuilds. Prints only on change. 3. Session filter is any-broker: sessions defined on their financial centres' civil clocks (London 08-16 Europe/London, NY 08-17 America/New_York, Tokyo 09-18 Asia/Tokyo), converted to UTC by each centre's own computed DST rule (EU last-Sun-Mar/Oct, US 2nd-Sun-Mar/1st-Sun-Nov), then to broker time by the MEASURED server-vs-GMT offset (half-hour brokers included). Windows may wrap midnight in broker time - the interval test handles it. Replaces the EET-hardcoded anchors, which were correct on exactly one broker and got Tokyo wrong by an hour each European summer. 4. The current-session-table-for-history caveat resolved by analysis: the bars bound the error - a too-late assumed close meets no bars (zero error), a too-early one truncates conservatively (<=1h, never optimistic, cannot manufacture edge). Documented at the site. 5. The ensemble deploy gate mirrors the direction policy: blocked-side fires are not fired bars (certified == traded), the zero-skill reference uses only ACHIEVABLE baselines (always-short is not a strategy a long-only book can run), and one-sidedness BY POLICY is not degeneracy - the two-sided requirement applies only when both sides are allowed. Sell predictions keep their other jobs (exit triggers, consensus dilution) untouched. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 12:14:01 -04:00
//--- ...and re-derive the Intelligent-direction drift verdict on the same cadence: the label
//--- cache it scans shifts with new bars, and a verdict that only refreshed at full rebuilds
//--- could sit stale for weeks (flagged in the 2026-08-19 review). Prints only on change.
RefreshDriftVerdict();
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
//--- Diagnostic recall snapshot for the periodic progress log further below - populated inside
//--- the m_oosSamples>0 recall-gate block when this era actually computes it; stays -1 ("n/a"
//--- in the log) on eras that don't (era 0, or a stopped/cap-hit era).
int logBuyRecallPct = -1, logSellRecallPct = -1, logNeutralRecallPct = -1;
//--- Balanced accuracy (macro-recall) this era, surfaced in the log so the metric the checkpoint
//--- is now selected on is visible - see m_bestBalancedOos. -1 ("n/a") on eras that don't score.
int logBalancedAccPct = -1;
feat(ai): rank checkpoints on directional precision, not balanced accuracy Balanced accuracy is maximized by exactly the model this system must never deploy. Measured frontier at fixed signal strength, base rate 6.1%: tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0% tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3% tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5% It rises monotonically as the model calls MORE and is right LESS, because two of its three terms are directional recalls that a call-everything model drives to ~95%, while the Neutral term it sacrifices counts for only a third. The 2026-07-29 run landed exactly there: balanced 58-64% while calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base rate. Only the per-class recall floor stopped those deploying - a guard doing the job the objective should have been doing - and that same guard also rejected the genuinely useful sparse-but-precise checkpoints. Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how many were right. That is what a trading edge is. Two anti-degenerate floors bracket it, since precision alone is trivially maximized by calling almost nothing: coverage must reach a fraction of the true directional base rate (derived, not configured - it adapts to any symbol/timeframe/label rule), and precision must at least beat that base rate. Against the same frontier the deploy order inverts from tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first) to tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage) Balanced accuracy is kept in the log as a diagnostic and marked as such, so a run where the two disagree - the signature of an over-caller - is visible at a glance. MinRecall no longer decides what ships; it now only drives the diagnostic recall line and is a candidate for removal. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
int logCoveragePct = -1;
int logDirPrecPct = -1;
fix: refuse invalid SL/TP, fix the unreachable deploy floor, scale the horizon Three defects found by reading the 2026-08-01 training logs, all of which only became visible because the relabel made the numbers mean something. 1. A STALE ENUM TRAINED FOUR MODELS ON THE WRONG TARGET. `OnInit: trade settings snapshot - SL_Mode=1 TP_Mode=-101` -101 was TP_PREV_SWING, deleted from TAKE_PROFIT_MODE on 2026-07-31 in 7eb48f5. MetaTrader does not validate a saved enum input against the enum's current members, so charts saved before that kept the old integer. BarrierMultiples()'s `if(tpMult <= 0.0) tpMult = slMult;` then quietly turned it into a 1:1 barrier, and all four topologies trained ~250 eras against a strategy nobody selected - while the log reported "target 1.00*ATR" as though it were configured. Since the relabel these two inputs ARE the label definition, so this is not a bad trade setting, it is a wrong dataset. ValidateBarrier- Inputs() now refuses to start (INIT_FAILED + Alert + an explicit fix) on any value that is not an enum member. Members are enumerated rather than range-checked because both enums are sparse and carry negative sentinels, so no min/max test can tell a legal value from a deleted one - which is the entire failure mode. The fallback survives as belt-and-braces but now announces itself: a fallback that cannot say it fired is indistinguishable from correct behaviour. 2. THE DEPLOYABILITY FLOOR BECAME MATHEMATICALLY UNREACHABLE. `tradeableOK` required `dirPrecPct >= baseRatePct`, where baseRatePct is Buy+Sell as a share of all bars. At the old exact-pivot target that was ~6%, so "beat the base rate" read as "beat chance" and the test looked sound. Triple-barrier labels put it at ~83%, so the gate now demanded 83% directional precision - impossible by construction. Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing safe to deploy" at a perfectly healthy 43-45% precision, with no checkpoint able to ship however good it got. Replaced with ZERO-SKILL precision, max(Buy,Sell)/allBars: exactly the score of the degenerate always-call-one-direction model this floor exists to reject. Correct at any base rate - ~43% on the current labels, ~3% on the old rare-pivot ones. The era line now prints "(chance N%, edge +Mpp)" beside the selection score, because 44% precision is excellent against a 3% chance level and worthless against a 43% one, and reading the first as the second is what made tonight's run look better than it was. 3. THE HORIZON IGNORED THE BARRIER GEOMETRY. ComputeBarrierHorizonBars() returned the median ZigZag leg, which measures how long a ~1 ATR move takes and says nothing about how long the CONFIGURED barrier needs. First-passage time out of [-m,+k] scales with m*k, so a 1:3 barrier takes ~3x as long as 1:1; the unscaled horizon would have timed out most 1:3 trades and pushed Neutral straight back up, re-creating the imbalance the relabel removes. Now multiplied by slMult*tpMult, calibrated against a real measurement rather than assumed: the accidental 1:1 run resolved at horizon 12 with only 16.7% timeouts, so the swing median is the right scale at m*k=1. Verifiable, not just asserted: the prebuild now counts barriers that ended on the VERTICAL barrier and reports them as a share of Neutral. Neutral conflates "timed out" with "stopped out" and only the first indicts the horizon. Both builds compile 0 errors / 0 warnings. Forces a retrain - correcting TP_Mode re-keys the fingerprint (|TB:1:-101 -> |TB:1:3), which is right: no existing model was trained on the intended target. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:30:49 -04:00
//--- Zero-skill precision for this era's label mix - see chancePrecPct. Logged beside the selection
//--- score because the raw precision number is meaningless without it: 44% is excellent against a
//--- 3% chance level and worthless against a 43% one, and the whole 2026-08-01 confusion was
//--- reading the first as if it were the second.
int logChancePrecPct = -1;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- Predicted-rate (of all OOS bars this era, how often the model called this class at all) and
//--- precision (of the calls it made, how many were right) for Buy/Sell - m_oosBuyPredicted/
//--- m_oosSellPredicted (see that member's declaration comment) were already being tracked for
//--- exactly this but never surfaced anywhere. A recall-only view can't tell "the model never once
//--- calls Sell" (predicted rate stuck at 0%) apart from "the model calls Sell plenty but always on
//--- the wrong bars" (predicted rate healthy, precision near 0%) - both show up identically as 0%
//--- Sell recall, but point at completely different problems (a suppressed/dead output vs. a
//--- miscalibrated decision boundary), so this splits them out.
int logBuyPredPct = -1, logSellPredPct = -1, logBuyPrecPct = -1, logSellPrecPct = -1;
//--- Live-fired precision (%) per direction this era - the precision on just the bars that cleared
//--- the confidence floor under the live/prior-corrected decision, i.e. what would actually trade.
int logBuyFiredPrecPct = -1, logSellFiredPrecPct = -1;
bool shouldLogProgress = false;
//--- era complete (ran out of bars) or a stop was requested mid-era
if(add_loop)
{
m_eraCount++;
m_erasSinceCooldown++;
//--- EMA shadow-weight deployment: blend the shadow a small step (SHADOW_WEIGHT_TAU) toward
//--- Net's just-updated weights, every era - see m_shadowNet's declaration comment. Must run
//--- here, inside the era loop, not just once at Train()-end: the whole point is damping the
//--- WITHIN-run oscillation (era-to-era whipsaw), which a single end-of-run blend would miss
//--- entirely.
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
EnsureShadowNet();
if(CheckPointer(m_shadowNet) != POINTER_INVALID)
m_shadowNet.BlendWeightsFrom(Net, SHADOW_WEIGHT_TAU);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- Status-label progress is invisible with no chart (headless/optimization runs), and even in
//--- visual mode a long training run can otherwise look "stuck" for a long time with no
//--- Journal output at all - log progress at most every ~5s (real wall-clock, not simulated
//--- time) so an operator can tell it's actively working, not hung. The actual Print() is
//--- deferred past the recall-gate block below (see logBuyRecallPct etc.) so this line can
//--- show per-class OOS recall - once OOS accuracy alone clears the target, recall is the
//--- most common thing still silently blocking convergence, and previously had no visibility
//--- outside of a regression event.
static uint lastProgressLogTick = 0;
uint nowTick = GetTickCount();
shouldLogProgress = (nowTick - lastProgressLogTick >= 5000);
if(shouldLogProgress)
lastProgressLogTick = nowTick;
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- Era cap. There used to be a second, much smaller cap here for throwaway auto-tune
//--- candidates; the filter tuner does not train candidates at all, so only the real one remains.
int effectiveEraCap = m_maxErasPerRun;
diag: report |dW| alongside d|W| per layer, and drop two stale log claims The per-era `dW/W` line measured the change in each layer's weight NORM. That statistic cannot separate "this layer only shrank under weight decay" from "this layer moved somewhere useful" - a rotation at constant norm and pure decay can print the same number. It matters right now: on SP500 H1 the LSTM layers print a near-constant ~1.05%/era that exactly equals their geometric norm decay over 318 eras (HYB lstm2 12.966 -> 0.755, monotone, never once up), while a sibling conv oscillates around a much slower drift. Norm-change can only hint at that. Now prints norm(d|W|% / |dW|%). Under decay alone the two are equal; any gradient component adds in quadrature to the second, so a learning layer shows the second clearly larger. Diagnostic only - no training behaviour changes, fingerprint untouched. Also removes two log lines that described machinery deleted in 397b0ea: the plateau ladder's terminal stage still claimed "after a warm restart AND full gamma anneal", and the CONVERGED line "across a warm restart and a full focal-gamma anneal". Both printed on every CONV/LSTM convergence today. Same failure mode as the oversampling line 397b0ea fixed: a log that describes what an older version would have done is confirming evidence for a false hypothesis. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 14:33:29 -04:00
//--- PLATEAU LADDER, terminal stage: training stopped improving and both escape attempts (two
//--- learning-rate warm restarts) failed to find anything better - see the ladder in the era-end block
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- below, which is what raised m_plateauStage this far and already logged why. This is the
//--- normal, expected way a run finishes now that there is no absolute accuracy target to hit:
//--- it trains until it genuinely stops getting better, then deploys its best checkpoint.
//--- Same mechanism as the operator's "No" answer at the era cap (see that branch's comments for
//--- why m_trainingComplete is set here and why m_trainingStopRequested deliberately is NOT):
//--- stop ends this era loop, FinalizeTrainRun() then restores and deploys the best checkpoint.
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//--- ENSEMBLE: the ladder and the gate are the ensemble's, so the trigger is its verdict
//--- (g_ensDeployApproved), not this member's own stage. Every member sees the same flag on
//--- its next Train() call and finalises within one era of the others, each restoring its own
//--- half of the JOINT checkpoint - see the ENSEMBLE DEPLOY GATE block.
bool deployNow = m_ensembleMember
? (g_ensDeployApproved && m_haveOosCheckpoint)
fix(plateau): the IS-error early stop was inert for every ensemble member 15 hours of training, and the stop that exists to END a run announced itself 1,299 consecutive times without ending anything: SP500 ConvLSTM IN-SAMPLE ERROR PLATEAU - not improved in 1297 / 1298 / 1299 eras (best 0.2689, now 0.3269) ... era 1396, 1397, 1398 SP500 LSTM 536 eras SP500 CONV 442 eras SP500 PAI 150 eras XAUUSD HYB 478 eras XAUUSD LSTM 296 eras XAUUSD CONV 366 eras CAUSE: it wrote its decision into m_plateauStage, and EnsembleEraVerdict mirrors the shared ladder onto every member - `mm.m_plateauStage = g_ensPlateauStage` - on EVERY era, purely so each member's status line shows the collective stage. A display mirror was silently overwriting a decision, so the stop re-armed and re-fired the next era, forever. This is the worst possible direction for this particular bug. Every one of those 1,299 eras was scored out of sample and joined the family the deploy gate corrects over (Sidak, g_ensCandidateEras). The stop's entire purpose is to make that family SMALLER; instead the run spent fifteen hours raising its own bar. - m_isErrorPlateaued: a one-way per-member latch, cleared only by a fresh run. Nothing in the ladder may reset it. The stop condition and the two solo deploy conditions read the latch, not the mirrored stage. - The orchestrator combines: EnsembleEraVerdict requires UNANIMITY across participating members (same participation test the era barrier uses, so an excluded or finished member cannot veto). One member still learning can still move the combined vote, and the vote is what the gate certifies. - Fed in as `dueStage = PLATEAU_STAGE_DEPLOY`, NOT written to g_ensPlateauStage. The block that actually ends the run sits under `dueStage > g_ensPlateauStage`, so assigning the stage directly makes that test false and the deploy never happens - the same inert-write shape as the bug being fixed. Caught before committing; raising dueStage carries it through the ladder's own path (warm restarts skipped, family-wise vote test, measurement screen, joint checkpoint) unchanged. - g_ensIsPlateauAnnounced: announce once per run, not once per era. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 13:09:26 -04:00
: ((m_plateauStage >= PLATEAU_STAGE_DEPLOY || m_isErrorPlateaued) && m_bestPassedRecall && m_haveOosCheckpoint
fix(consistency): one geometry authority, one exit authority, and the MI screen finally gets a veto Consistency pass before a fresh deployment. Three places where two systems were choosing the same thing and one of them silently lost. 1. THE GEOMETRY SCAN'S DECISION WAS INERT - measured, not suspected. USDJPY, 2026-08-17: 14:24:12.844 adopting barrier geometry 2:8 ... Relabelling and training on it. 14:24:12.979 triple-barrier labels - stop 1.61*ATR, target 3.21*ATR ... this is what trains It adopted 2:8 and trained on 1.61:3.21. ReportBarrierGeometryScan wrote only m_sl_mode/m_tp_mode, and BarrierMultiples ranks the DERIVED pair ABOVE those ints - so on any model carrying a derived pair (every model with a .cfg, including a fresh one whose weights are gone but whose sidecar survived) the adoption changed nothing. Worse, had it changed something it would have been undone immediately: the adoption sets m_labelCachePrebuilt = false, and that prebuild re-runs DeriveBarrierGeometry at era 0, which overwrites m_derivedSl/TpMult from the excursion quantiles. ONE AUTHORITY: the derived pair, because it is what the labels read, what the deploy gate certifies, what g_Derived*AtrMult places on the live order, and what the .cfg pins across restarts. The scan now writes THAT (floored by MIN_SL_ATR_MULTIPLIER, the same floor DeriveBarrierGeometry applies so the live stop can never be wider than the labelled one), republishes to the bridge immediately rather than at the next era end, and forces the sidecar to be rewritten. m_geometryAdopted latches it so the derive pass the adoption itself triggers cannot overwrite it. The scan outranks the derive for an evidential reason, not an architectural one: its winner cleared a permutation test against the null of the MAXIMUM over every eligible pairing, and it scores the incumbent derived pair as a peer in that same field. The derive is a descriptive quantile read with no significance test attached. BEHAVIOURAL CHANGE, and the reason to flag it before a fresh test: barrier geometry will now actually move when the scan says so. Until today it never did. 2. ONE EXIT AUTHORITY, tied to whose certificate the trade was placed under. CheckClosePosition had two routes. The AI early-exit reads the AI vote undiluted and is exactly what the new exit replay reproduces. The blended route thresholds m_direction, the average over EVERY filter including classic ones whose live votes pass 3 never computes - so it can close a position the certificate never modelled, and no replay can ever check it. When g_DerivedSlAtrMult > 0 the AI's measured geometry is on the order, which means the deploy gate's certificate is the reason the trade exists. In that state the AI now governs the exit and the blended route is suppressed. Classic-only configurations are untouched: there the blended route is the only exit opinion and stays exactly as it was. Nothing moves at the shipped defaults either way (Min_Vote_Close = Disabled). 3. EDGEFINDER, SECOND HALF: THE MEASUREMENT NOW STEERS. The MI suite has always printed its verdicts and then trained the direction target regardless of what they said. That gap IS the difference between this and the EdgeFinder discipline: measure what the market offers, THEN aim. m_dirEvidence is set when EITHER the feature/label mutual information OR the normalised excursion asymmetry clears its block-permuted null - an OR, because the two look for the same thing by different routes and requiring both would reject on the weaker of two independent measurements. Normalised asymmetry specifically, never the raw one, which is the volatility confound. Deploy - solo AND ensemble - now requires it. A run without it still trains, and keeps its checkpoint: the research value is real and the measurement can be wrong. It simply may not go live. Reported separately from the statistical gate because the remedy is different: a failed selection test says train differently, this says look somewhere else. Excursion SIZE keeps clearing where direction does not, and that is a risk-control head rather than an entry signal. For the ensemble the check is per-chart by construction - the MI suite runs once and shares its outcome across members - which is the honest treatment: four models finding nothing between them is not four chances at an edge, it is four fits to the same absent information. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:43:20 -04:00
//--- AND the market was measured to hold directional information in the
//--- first place. See m_dirEvidence: the MI suite has always printed this
//--- verdict and then deployed regardless of what it said.
&& m_dirEvidence);
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
if(deployNow)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
stop = true;
m_trainingComplete = true;
}
else if(effectiveEraCap > 0 && m_erasSinceCooldown >= effectiveEraCap)
{
//--- Era cap reached without converging: ask the operator whether to keep training or
//--- deploy the best checkpoint and stop (see PromptContinuePastEraCap / m_maxErasPerRun).
if(PromptContinuePastEraCap(dOosForecast))
{
m_erasSinceCooldown = 0; // keep training - reset the cap window
feat(ai): measure precision per confidence tier; fix stale metric labels Two things the 2026-07-30 run exposed. 1. Every user-facing message still called the selection metric "balanced accuracy". It has ranked on directional precision since a142749, so "CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5% PRECISION as if it were macro-recall, while the same era logged an actual balanced accuracy of 49%. Two different numbers under one name, in the line that announces a deploy. Relabelled at every site, including the stage-3 refusal, which still described the per-class recall floor that stopped being the gate. 2. Precision is now bucketed by confidence tier and logged per era, both per-tier and cumulatively from each tier upward: | tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ... The per-tier number says whether confidence is calibrated to correctness at all; if it does not rise T0->T3, raising the floor buys nothing and that is the finding. The ">=" number is what a floor would actually deliver, with its fire count, so the coverage cost is visible in the same line. Tier weights are 25/50/75/100, so for an AI-only config Min_Vote_Open maps straight across: 50 = ">=T1", 75 = ">=T2", 100 = ">=T3". Bucketing happens at the existing live-fired accounting site, so it measures exactly the population that trades - not the raw argmax. Both builds compile 0 errors, 0 warnings. No retrain needed for either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
Print(ID + ": hit the " + IntegerToString(m_maxErasPerRun) + "-era cap (best dir-precision " + DoubleToString(m_bestBalancedOos, 1) + "%, blended OOS " + DoubleToString(dOosForecast, 1) + "%) - CONTINUING training by operator choice.");
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
else
{
//--- stop: end THIS era loop now; FinalizeTrainRun (reached via the stop path below,
//--- because stop==true) deploys the best checkpoint. Deliberately do NOT set
//--- m_trainingStopRequested here: m_trainingComplete alone already routes every later
//--- tick to RefreshConvergedSignal (see ScheduleTrainingIfNeeded's if-branch precedence),
//--- so training never re-arms - and leaving m_trainingStopRequested false lets the deployed
//--- model run live inference AND online continual learning IN-SESSION, exactly like a
//--- normal-convergence deploy (which never sets it either). A panel Stop (StopTraining())
//--- still sets it and halts everything, including online learning - that distinction is
//--- preserved. See OnlineLearnStep()'s gate.
stop = true;
//--- Operator DELIBERATELY chose to deploy this best checkpoint as the final model. That's a
//--- terminal decision and must be PERSISTED as such: mark it complete so a later reload
//--- (chart restart OR strategy tester) runs inference instead of silently resuming a full
//--- training run. This is the terminal-deploy path; a mid-training Stop click
//--- (StopTraining()) leaves m_trainingComplete false on purpose so that genuinely-
//--- interrupted run does resume. Note the m_trainingComplete=(m_objectiveMet&&m_oosStable)
//--- line below is inside if(!stop), so it can't clobber this back to false on this path.
m_trainingComplete = true;
feat(ai): measure precision per confidence tier; fix stale metric labels Two things the 2026-07-30 run exposed. 1. Every user-facing message still called the selection metric "balanced accuracy". It has ranked on directional precision since a142749, so "CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5% PRECISION as if it were macro-recall, while the same era logged an actual balanced accuracy of 49%. Two different numbers under one name, in the line that announces a deploy. Relabelled at every site, including the stage-3 refusal, which still described the per-class recall floor that stopped being the gate. 2. Precision is now bucketed by confidence tier and logged per era, both per-tier and cumulatively from each tier upward: | tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ... The per-tier number says whether confidence is calibrated to correctness at all; if it does not rise T0->T3, raising the floor buys nothing and that is the finding. The ">=" number is what a floor would actually deliver, with its fire count, so the coverage cost is visible in the same line. Tier weights are 25/50/75/100, so for an AI-only config Min_Vote_Open maps straight across: 50 = ">=T1", 75 = ">=T2", 100 = ">=T3". Bucketing happens at the existing live-fired accounting site, so it measures exactly the population that trades - not the raw argmax. Both builds compile 0 errors, 0 warnings. No retrain needed for either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
Print(ID + ": hit the " + IntegerToString(m_maxErasPerRun) + "-era cap before the plateau ladder finished (best dir-precision " + DoubleToString(m_bestBalancedOos, 1) + "%, blended OOS " + DoubleToString(dOosForecast, 1) + "%) - operator chose to DEPLOY the best checkpoint as final (marked complete; reloads will run inference, not retrain). Reaching this cap now means the run was still finding new bests, or never cleared the per-class recall floor (need >=" + IntegerToString(m_minDirectionalRecallPct) + "% each) - raise the era cap for the former, relax MinRecall/SwingConfirmationBars for the latter.");
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
//--- NOT blocked - this branch is an explicit operator decision and stays one. But the
//--- automatic ladder would refuse this model, so say so plainly rather than letting the
//--- deploy read as a clean pass. See DEPLOY_FAMILY_WISE_ALPHA.
ReportSelectionGateVerdict("era-cap deploy");
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
}
}
if(!stop)
{
dError = Net.getRecentAverageError();
if(add_loop)
{
if(m_oosSamples > 0)
{
// Confidence calibration (classification head only - see m_confidenceCalScale's
// declaration comment): compare this era's actual OOS accuracy against the average
// confidence magnitude the model claimed, EMA-blend the resulting scale into
// m_confidenceCalScale so SignedAIConfidence() reports something closer to a real
// probability instead of the raw, uncalibrated softmax value.
if(m_outputNeuronsCount == 3 && m_oosConfidenceSum > 0.0)
{
double empiricalAccuracy = (double)(m_oosBuyHits + m_oosSellHits + m_oosNeutralHits) / m_oosSamples;
double avgClaimedConfidence = m_oosConfidenceSum / m_oosSamples;
double eraScale = MathMax(0.3, MathMin(1.5, empiricalAccuracy / avgClaimedConfidence));
m_confidenceCalScale += (eraScale - m_confidenceCalScale) / Net.recentAverageSmoothingFactor;
}
// Per-class recall gate, symmetric across all three classes: a model that "wins" on
// blended dOosForecast purely by calling everything Neutral (or, just as biased, by
// over-calling Buy/Sell at Neutral's expense) would still pass a plain accuracy check -
// require Buy, Sell, AND Neutral OOS recall to each individually clear
// m_minDirectionalRecallPct so the network can't converge while biased toward any one
// output. A class with FEWER than MIN_OOS_CLASS_SAMPLES_FOR_GATE true OOS samples this
// era doesn't block (recallPct == -1 => treated as passing) so a thin OOS window doesn't
// deadlock convergence early in a run. Computed BEFORE the checkpoint/eta-decay block
// below (not just the final m_objectiveMet gate) so "best" ranking is recall-aware too -
// see isBetterEra's comment for why that matters.
//
// The threshold matters: a bare ">0" here (the original behavior) let a run converge at
// era 44-46 with the OOS window containing exactly ZERO true Buy/Sell bars that era
// (logged as "OOS recall Buy:n/a Sell:n/a Neutral:100%") - a full Neutral-only collapse
// that the gate waved through because there was nothing to measure recall against, not
// because the model was actually unbiased. Requiring a real minimum sample count means
// an unlucky/thin OOS slice blocks convergence instead of silently passing it.
int buyRecallPct = (m_oosBuyTotal >= MIN_OOS_CLASS_SAMPLES_FOR_GATE) ? (int)MathRound(100.0 * m_oosBuyHits / m_oosBuyTotal) : -1;
int sellRecallPct = (m_oosSellTotal >= MIN_OOS_CLASS_SAMPLES_FOR_GATE) ? (int)MathRound(100.0 * m_oosSellHits / m_oosSellTotal) : -1;
int neutralRecallPct = (m_oosNeutralTotal >= MIN_OOS_CLASS_SAMPLES_FOR_GATE) ? (int)MathRound(100.0 * m_oosNeutralHits / m_oosNeutralTotal) : -1;
logBuyRecallPct = buyRecallPct;
logSellRecallPct = sellRecallPct;
logNeutralRecallPct = neutralRecallPct;
m_lastBuyRecallPct = buyRecallPct;
m_lastSellRecallPct = sellRecallPct;
// Predicted-rate (share of ALL OOS bars this era the model called this class, regardless
// of whether that call was right) and precision (of just those calls, how many were
// right) - see logBuyPredPct's declaration comment above for why this is worth logging
// alongside recall. Denominator is the per-era OOS bar count (sum of the per-class true
// totals, all tallied in the same pass-3 block and reset together each era) - NOT
// m_oosSamples, which only resets on a full model reset and so accumulates across every
// era of the run: dividing this era's calls by that all-run total diluted the logged
// rate by roughly the era number (observed: era-15 "Buy:2%" that was really ~30%),
// making a genuinely directional model read as a nearly-dead output.
int oosEraBars = m_oosBuyTotal + m_oosSellTotal + m_oosNeutralTotal;
logBuyPredPct = (oosEraBars > 0) ? (int)MathRound(100.0 * m_oosBuyPredicted / oosEraBars) : -1;
logSellPredPct = (oosEraBars > 0) ? (int)MathRound(100.0 * m_oosSellPredicted / oosEraBars) : -1;
logBuyPrecPct = (m_oosBuyPredicted > 0) ? (int)MathRound(100.0 * m_oosBuyPredictedHits / m_oosBuyPredicted) : -1;
logSellPrecPct = (m_oosSellPredicted > 0) ? (int)MathRound(100.0 * m_oosSellPredictedHits / m_oosSellPredicted) : -1;
//--- Live-fired precision (what actually trades - see m_oosBuyFired): of the directional
//--- calls that cleared the confidence floor under the live/prior-corrected rule this era,
//--- how many were right. Cached for the panel/log; -1 = the model fired none this era.
logBuyFiredPrecPct = (m_oosBuyFired > 0) ? (int)MathRound(100.0 * m_oosBuyFiredHits / m_oosBuyFired) : -1;
logSellFiredPrecPct = (m_oosSellFired > 0) ? (int)MathRound(100.0 * m_oosSellFiredHits / m_oosSellFired) : -1;
m_lastBuyFiredPrecPct = logBuyFiredPrecPct;
m_lastSellFiredPrecPct = logSellFiredPrecPct;
m_lastBuyFired = m_oosBuyFired;
m_lastSellFired = m_oosSellFired;
feat(ai): rank checkpoints on directional precision, not balanced accuracy Balanced accuracy is maximized by exactly the model this system must never deploy. Measured frontier at fixed signal strength, base rate 6.1%: tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0% tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3% tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5% It rises monotonically as the model calls MORE and is right LESS, because two of its three terms are directional recalls that a call-everything model drives to ~95%, while the Neutral term it sacrifices counts for only a third. The 2026-07-29 run landed exactly there: balanced 58-64% while calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base rate. Only the per-class recall floor stopped those deploying - a guard doing the job the objective should have been doing - and that same guard also rejected the genuinely useful sparse-but-precise checkpoints. Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how many were right. That is what a trading edge is. Two anti-degenerate floors bracket it, since precision alone is trivially maximized by calling almost nothing: coverage must reach a fraction of the true directional base rate (derived, not configured - it adapts to any symbol/timeframe/label rule), and precision must at least beat that base rate. Against the same frontier the deploy order inverts from tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first) to tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage) Balanced accuracy is kept in the log as a diagnostic and marked as such, so a run where the two disagree - the signature of an over-caller - is visible at a glance. MinRecall no longer decides what ships; it now only drives the diagnostic recall line and is a candidate for removal. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
//--- SELECTION METRIC. Ranking moved off balanced accuracy (macro-recall) 2026-07-30
//--- because that metric is maximized by exactly the model this system must never deploy.
//--- Measured frontier at a fixed signal strength, base rate 6.1%:
//--- tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0%
//--- tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3%
//--- tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5%
//--- Balanced accuracy rises monotonically as the model calls MORE and is right LESS,
//--- because two of its three terms are directional recalls that a call-everything model
//--- drives to ~95% while the Neutral term it sacrifices counts for only a third. The
//--- 2026-07-29 run landed exactly there: balanced 58-64% while calling a direction on
//--- ~100% of bars at a 5-7% win rate against a ~6% base rate - no information at all.
//--- Only the per-class recall floor stopped those from deploying, i.e. a guard was doing
//--- the job the objective should have been doing, and the same guard also rejected the
//--- genuinely useful sparse-but-precise checkpoints (directional recall 4-6%).
//--- Ranking is now DIRECTIONAL PRECISION - of the bars this model called Buy or Sell,
//--- how many were right - which is what a trading edge actually is. Two anti-degenerate
//--- floors bracket it, because precision alone is trivially maximized by calling almost
//--- nothing: coverage must reach a fraction of the true base rate, and precision must at
//--- least beat that base rate (a model no better than the coin is not an edge).
fix: the deploy gate graded the un-thresholded model coveragePct, dirPrecPct and the declustered TRADED tally were all computed from oPrevSignal - the RAW argmax - while the live order, the arrow and the panel all run on oDeploySignal, which is argmax AFTER the confidence threshold. The gate was certifying a strategy the EA does not trade. Invisible until now: the threshold sat at ~0.02, so the two populations were the same set. The held-out calibration slice (2189316) moved it to 0.14-0.40 and the gap opened immediately - PAI era 256 graded 100% coverage while its traded population was 21% (3,399 of ~16,200 OOS bars). Consequences that were being hidden: - coveragePct >= minCoveragePct was tested against the wrong population, so a model whose TRADED coverage falls under the 24.8% floor still read as clearing it - precSE = sqrt(p(1-p)/n) used n ~16,000 instead of n ~3,400, so the EDGE_MIN_SIGMAS bar was ~2.2x too lenient on the real evidence - the NMS replay declustered a different, larger stream than live, so threshold-rejected bars consumed cluster slots and set alternation state Gate quantities now read m_oosBuyFired/m_oosSellFired (the thresholded population, already tracked for the live-precision line) and the NMS replay runs on oDeploySignal. The threshold can only turn a direction into Neutral, never flip a side, so the fired set is a strict subset and every per-bar outcome is the one already computed. Recall and logBuyPrecPct deliberately stay on the raw argmax: they measure intrinsic class separation, and thresholding them would conflate "cannot separate the classes" with "declines to act on the separation it found". This is the 9a7c37f defect class, and the NMS block carried a comment warning about it while committing it three lines above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 19:34:32 -04:00
//--- THE POPULATION THAT ACTUALLY TRADES (m_oosBuyFired - the calls surviving the
//--- confidence threshold), not the raw argmax (m_oosBuyPredicted). Those two were the
//--- same set for as long as the threshold sat near zero, so the distinction cost nothing
//--- and the gate read the argmax. The held-out calibration slice ended that: thresholds
//--- moved from ~0.02 to 0.14-0.40, and on 2026-08-10 PAI era 256 the argmax population
//--- was 100% of bars while the traded population was 21% - so the gate was certifying a
//--- trade-every-bar strategy that the EA does not run. AdjustedSignalFromSoftmax() gates
//--- the live order, the arrow and the panel; it has to gate the deployment decision too.
//--- This is the 9a7c37f defect class (gate certifies one thing, execution does another),
//--- and the NMS block below carried a comment warning about it while committing it.
//---
//--- The threshold can only turn a direction into Neutral, never flip Buy to Sell, so the
//--- fired set is a strict subset of the argmax set and every per-bar outcome (oWinLong/
//--- oWinShort) is the one already computed for that bar.
//---
//--- Deliberately NOT changed alongside: the recall figures and logBuyPrecPct, which stay
//--- on the raw argmax. Those measure the model's intrinsic class separation - a
//--- diagnostic of whether it is learning at all - and thresholding them would conflate
//--- "cannot separate the classes" with "declines to act on the separation it found".
int oosDirCalls = m_oosBuyFired + m_oosSellFired;
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
//--- WINS, not label agreement - see m_oosBuyPredictedWins for the full argument. The
//--- label-agreement figure is still computed and still logged (logBuyPrecPct/
//--- logSellPrecPct), because it is the right diagnostic for class separation; it is just
//--- not the right thing to gate a DEPLOYMENT on, which is a question about money.
fix: the deploy gate graded the un-thresholded model coveragePct, dirPrecPct and the declustered TRADED tally were all computed from oPrevSignal - the RAW argmax - while the live order, the arrow and the panel all run on oDeploySignal, which is argmax AFTER the confidence threshold. The gate was certifying a strategy the EA does not trade. Invisible until now: the threshold sat at ~0.02, so the two populations were the same set. The held-out calibration slice (2189316) moved it to 0.14-0.40 and the gap opened immediately - PAI era 256 graded 100% coverage while its traded population was 21% (3,399 of ~16,200 OOS bars). Consequences that were being hidden: - coveragePct >= minCoveragePct was tested against the wrong population, so a model whose TRADED coverage falls under the 24.8% floor still read as clearing it - precSE = sqrt(p(1-p)/n) used n ~16,000 instead of n ~3,400, so the EDGE_MIN_SIGMAS bar was ~2.2x too lenient on the real evidence - the NMS replay declustered a different, larger stream than live, so threshold-rejected bars consumed cluster slots and set alternation state Gate quantities now read m_oosBuyFired/m_oosSellFired (the thresholded population, already tracked for the live-precision line) and the NMS replay runs on oDeploySignal. The threshold can only turn a direction into Neutral, never flip a side, so the fired set is a strict subset and every per-bar outcome is the one already computed. Recall and logBuyPrecPct deliberately stay on the raw argmax: they measure intrinsic class separation, and thresholding them would conflate "cannot separate the classes" with "declines to act on the separation it found". This is the 9a7c37f defect class, and the NMS block carried a comment warning about it while committing it three lines above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 19:34:32 -04:00
int oosDirHits = m_oosBuyFiredHits + m_oosSellFiredHits;
feat(ai): rank checkpoints on directional precision, not balanced accuracy Balanced accuracy is maximized by exactly the model this system must never deploy. Measured frontier at fixed signal strength, base rate 6.1%: tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0% tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3% tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5% It rises monotonically as the model calls MORE and is right LESS, because two of its three terms are directional recalls that a call-everything model drives to ~95%, while the Neutral term it sacrifices counts for only a third. The 2026-07-29 run landed exactly there: balanced 58-64% while calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base rate. Only the per-class recall floor stopped those deploying - a guard doing the job the objective should have been doing - and that same guard also rejected the genuinely useful sparse-but-precise checkpoints. Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how many were right. That is what a trading edge is. Two anti-degenerate floors bracket it, since precision alone is trivially maximized by calling almost nothing: coverage must reach a fraction of the true directional base rate (derived, not configured - it adapts to any symbol/timeframe/label rule), and precision must at least beat that base rate. Against the same frontier the deploy order inverts from tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first) to tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage) Balanced accuracy is kept in the log as a diagnostic and marked as such, so a run where the two disagree - the signature of an over-caller - is visible at a glance. MinRecall no longer decides what ships; it now only drives the diagnostic recall line and is a candidate for removal. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
int oosDirTrue = m_oosBuyTotal + m_oosSellTotal;
bool coverageMeasurable = (oosEraBars > 0 && oosDirTrue > 0);
double coveragePct = coverageMeasurable ? 100.0 * oosDirCalls / oosEraBars : -1.0;
double baseRatePct = coverageMeasurable ? 100.0 * oosDirTrue / oosEraBars : -1.0;
double dirPrecPct = (oosDirCalls > 0) ? 100.0 * oosDirHits / oosDirCalls : -1.0;
double minCoveragePct = coverageMeasurable ? baseRatePct * MIN_COVERAGE_FRACTION_OF_BASE_RATE : -1.0;
fix: refuse invalid SL/TP, fix the unreachable deploy floor, scale the horizon Three defects found by reading the 2026-08-01 training logs, all of which only became visible because the relabel made the numbers mean something. 1. A STALE ENUM TRAINED FOUR MODELS ON THE WRONG TARGET. `OnInit: trade settings snapshot - SL_Mode=1 TP_Mode=-101` -101 was TP_PREV_SWING, deleted from TAKE_PROFIT_MODE on 2026-07-31 in 7eb48f5. MetaTrader does not validate a saved enum input against the enum's current members, so charts saved before that kept the old integer. BarrierMultiples()'s `if(tpMult <= 0.0) tpMult = slMult;` then quietly turned it into a 1:1 barrier, and all four topologies trained ~250 eras against a strategy nobody selected - while the log reported "target 1.00*ATR" as though it were configured. Since the relabel these two inputs ARE the label definition, so this is not a bad trade setting, it is a wrong dataset. ValidateBarrier- Inputs() now refuses to start (INIT_FAILED + Alert + an explicit fix) on any value that is not an enum member. Members are enumerated rather than range-checked because both enums are sparse and carry negative sentinels, so no min/max test can tell a legal value from a deleted one - which is the entire failure mode. The fallback survives as belt-and-braces but now announces itself: a fallback that cannot say it fired is indistinguishable from correct behaviour. 2. THE DEPLOYABILITY FLOOR BECAME MATHEMATICALLY UNREACHABLE. `tradeableOK` required `dirPrecPct >= baseRatePct`, where baseRatePct is Buy+Sell as a share of all bars. At the old exact-pivot target that was ~6%, so "beat the base rate" read as "beat chance" and the test looked sound. Triple-barrier labels put it at ~83%, so the gate now demanded 83% directional precision - impossible by construction. Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing safe to deploy" at a perfectly healthy 43-45% precision, with no checkpoint able to ship however good it got. Replaced with ZERO-SKILL precision, max(Buy,Sell)/allBars: exactly the score of the degenerate always-call-one-direction model this floor exists to reject. Correct at any base rate - ~43% on the current labels, ~3% on the old rare-pivot ones. The era line now prints "(chance N%, edge +Mpp)" beside the selection score, because 44% precision is excellent against a 3% chance level and worthless against a 43% one, and reading the first as the second is what made tonight's run look better than it was. 3. THE HORIZON IGNORED THE BARRIER GEOMETRY. ComputeBarrierHorizonBars() returned the median ZigZag leg, which measures how long a ~1 ATR move takes and says nothing about how long the CONFIGURED barrier needs. First-passage time out of [-m,+k] scales with m*k, so a 1:3 barrier takes ~3x as long as 1:1; the unscaled horizon would have timed out most 1:3 trades and pushed Neutral straight back up, re-creating the imbalance the relabel removes. Now multiplied by slMult*tpMult, calibrated against a real measurement rather than assumed: the accidental 1:1 run resolved at horizon 12 with only 16.7% timeouts, so the swing median is the right scale at m*k=1. Verifiable, not just asserted: the prebuild now counts barriers that ended on the VERTICAL barrier and reports them as a share of Neutral. Neutral conflates "timed out" with "stopped out" and only the first indicts the horizon. Both builds compile 0 errors / 0 warnings. Forces a retrain - correcting TP_Mode re-keys the fingerprint (|TB:1:-101 -> |TB:1:3), which is right: no existing model was trained on the intended target. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:30:49 -04:00
//--- ZERO-SKILL PRECISION: what a model with no information scores on this metric, by
//--- always calling whichever direction is more common. Its precision is that class's
//--- share of ALL bars, because the bars it calls are uncorrelated with the labels.
//--- This REPLACED `dirPrecPct >= baseRatePct` on 2026-08-01, which was wrong the moment
//--- the labels stopped being rare. baseRatePct is Buy+Sell as a share of all bars: at the
//--- old exact-pivot target that was ~6%, so "beat the base rate" read as "beat chance"
//--- and the test looked sound. Triple-barrier labels put it at ~83%, and the gate then
//--- demanded 83% directional precision - unreachable by construction, so NOTHING could
//--- ever deploy. Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing
//--- safe to deploy" at a genuinely healthy 43-45% precision.
//--- max(Buy,Sell) is the right benchmark at ANY base rate: it is exactly the score of the
//--- degenerate always-call-one-direction model this floor exists to reject, and it
//--- degrades correctly to ~3% on the old rare-pivot labels.
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
//---
//--- COUNTED IN WINS since 2026-08-09, matching dirPrecPct above. The always-Buy model is
//--- scored the way the real model now is: how often its trade PAID, which is
//--- m_oosWinLongTotal / all scored bars - not how often the collapsed label happened to
//--- read Buy. Those diverged the moment the measured geometry put the target nearer than
//--- the stop: label-Buy was 37.5% of bars while a long actually won on ~67% of them, so
//--- the gate was benchmarking a win rate against a label frequency and clearing models
//--- 30pp short of break-even. Now chance and break-even coincide again by construction -
//--- an always-long model wins m/(m+k), which IS the break-even rate for a k:m trade - so
//--- clearing this reference by EDGE_MIN_SIGMAS means positive expectancy and nothing else.
fix: refuse invalid SL/TP, fix the unreachable deploy floor, scale the horizon Three defects found by reading the 2026-08-01 training logs, all of which only became visible because the relabel made the numbers mean something. 1. A STALE ENUM TRAINED FOUR MODELS ON THE WRONG TARGET. `OnInit: trade settings snapshot - SL_Mode=1 TP_Mode=-101` -101 was TP_PREV_SWING, deleted from TAKE_PROFIT_MODE on 2026-07-31 in 7eb48f5. MetaTrader does not validate a saved enum input against the enum's current members, so charts saved before that kept the old integer. BarrierMultiples()'s `if(tpMult <= 0.0) tpMult = slMult;` then quietly turned it into a 1:1 barrier, and all four topologies trained ~250 eras against a strategy nobody selected - while the log reported "target 1.00*ATR" as though it were configured. Since the relabel these two inputs ARE the label definition, so this is not a bad trade setting, it is a wrong dataset. ValidateBarrier- Inputs() now refuses to start (INIT_FAILED + Alert + an explicit fix) on any value that is not an enum member. Members are enumerated rather than range-checked because both enums are sparse and carry negative sentinels, so no min/max test can tell a legal value from a deleted one - which is the entire failure mode. The fallback survives as belt-and-braces but now announces itself: a fallback that cannot say it fired is indistinguishable from correct behaviour. 2. THE DEPLOYABILITY FLOOR BECAME MATHEMATICALLY UNREACHABLE. `tradeableOK` required `dirPrecPct >= baseRatePct`, where baseRatePct is Buy+Sell as a share of all bars. At the old exact-pivot target that was ~6%, so "beat the base rate" read as "beat chance" and the test looked sound. Triple-barrier labels put it at ~83%, so the gate now demanded 83% directional precision - impossible by construction. Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing safe to deploy" at a perfectly healthy 43-45% precision, with no checkpoint able to ship however good it got. Replaced with ZERO-SKILL precision, max(Buy,Sell)/allBars: exactly the score of the degenerate always-call-one-direction model this floor exists to reject. Correct at any base rate - ~43% on the current labels, ~3% on the old rare-pivot ones. The era line now prints "(chance N%, edge +Mpp)" beside the selection score, because 44% precision is excellent against a 3% chance level and worthless against a 43% one, and reading the first as the second is what made tonight's run look better than it was. 3. THE HORIZON IGNORED THE BARRIER GEOMETRY. ComputeBarrierHorizonBars() returned the median ZigZag leg, which measures how long a ~1 ATR move takes and says nothing about how long the CONFIGURED barrier needs. First-passage time out of [-m,+k] scales with m*k, so a 1:3 barrier takes ~3x as long as 1:1; the unscaled horizon would have timed out most 1:3 trades and pushed Neutral straight back up, re-creating the imbalance the relabel removes. Now multiplied by slMult*tpMult, calibrated against a real measurement rather than assumed: the accidental 1:1 run resolved at horizon 12 with only 16.7% timeouts, so the swing median is the right scale at m*k=1. Verifiable, not just asserted: the prebuild now counts barriers that ended on the VERTICAL barrier and reports them as a share of Neutral. Neutral conflates "timed out" with "stopped out" and only the first indicts the horizon. Both builds compile 0 errors / 0 warnings. Forces a retrain - correcting TP_Mode re-keys the fingerprint (|TB:1:-101 -> |TB:1:3), which is right: no existing model was trained on the intended target. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:30:49 -04:00
double chancePrecPct = coverageMeasurable
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
? 100.0 * MathMax(m_oosWinLongTotal, m_oosWinShortTotal) / oosEraBars : -1.0;
feat(ai): rank checkpoints on directional precision, not balanced accuracy Balanced accuracy is maximized by exactly the model this system must never deploy. Measured frontier at fixed signal strength, base rate 6.1%: tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0% tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3% tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5% It rises monotonically as the model calls MORE and is right LESS, because two of its three terms are directional recalls that a call-everything model drives to ~95%, while the Neutral term it sacrifices counts for only a third. The 2026-07-29 run landed exactly there: balanced 58-64% while calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base rate. Only the per-class recall floor stopped those deploying - a guard doing the job the objective should have been doing - and that same guard also rejected the genuinely useful sparse-but-precise checkpoints. Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how many were right. That is what a trading edge is. Two anti-degenerate floors bracket it, since precision alone is trivially maximized by calling almost nothing: coverage must reach a fraction of the true directional base rate (derived, not configured - it adapts to any symbol/timeframe/label rule), and precision must at least beat that base rate. Against the same frontier the deploy order inverts from tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first) to tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage) Balanced accuracy is kept in the log as a diagnostic and marked as such, so a run where the two disagree - the signature of an over-caller - is visible at a glance. MinRecall no longer decides what ships; it now only drives the diagnostic recall line and is a candidate for removal. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
logCoveragePct = (int)MathRound(coveragePct);
logDirPrecPct = (int)MathRound(dirPrecPct);
fix: refuse invalid SL/TP, fix the unreachable deploy floor, scale the horizon Three defects found by reading the 2026-08-01 training logs, all of which only became visible because the relabel made the numbers mean something. 1. A STALE ENUM TRAINED FOUR MODELS ON THE WRONG TARGET. `OnInit: trade settings snapshot - SL_Mode=1 TP_Mode=-101` -101 was TP_PREV_SWING, deleted from TAKE_PROFIT_MODE on 2026-07-31 in 7eb48f5. MetaTrader does not validate a saved enum input against the enum's current members, so charts saved before that kept the old integer. BarrierMultiples()'s `if(tpMult <= 0.0) tpMult = slMult;` then quietly turned it into a 1:1 barrier, and all four topologies trained ~250 eras against a strategy nobody selected - while the log reported "target 1.00*ATR" as though it were configured. Since the relabel these two inputs ARE the label definition, so this is not a bad trade setting, it is a wrong dataset. ValidateBarrier- Inputs() now refuses to start (INIT_FAILED + Alert + an explicit fix) on any value that is not an enum member. Members are enumerated rather than range-checked because both enums are sparse and carry negative sentinels, so no min/max test can tell a legal value from a deleted one - which is the entire failure mode. The fallback survives as belt-and-braces but now announces itself: a fallback that cannot say it fired is indistinguishable from correct behaviour. 2. THE DEPLOYABILITY FLOOR BECAME MATHEMATICALLY UNREACHABLE. `tradeableOK` required `dirPrecPct >= baseRatePct`, where baseRatePct is Buy+Sell as a share of all bars. At the old exact-pivot target that was ~6%, so "beat the base rate" read as "beat chance" and the test looked sound. Triple-barrier labels put it at ~83%, so the gate now demanded 83% directional precision - impossible by construction. Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing safe to deploy" at a perfectly healthy 43-45% precision, with no checkpoint able to ship however good it got. Replaced with ZERO-SKILL precision, max(Buy,Sell)/allBars: exactly the score of the degenerate always-call-one-direction model this floor exists to reject. Correct at any base rate - ~43% on the current labels, ~3% on the old rare-pivot ones. The era line now prints "(chance N%, edge +Mpp)" beside the selection score, because 44% precision is excellent against a 3% chance level and worthless against a 43% one, and reading the first as the second is what made tonight's run look better than it was. 3. THE HORIZON IGNORED THE BARRIER GEOMETRY. ComputeBarrierHorizonBars() returned the median ZigZag leg, which measures how long a ~1 ATR move takes and says nothing about how long the CONFIGURED barrier needs. First-passage time out of [-m,+k] scales with m*k, so a 1:3 barrier takes ~3x as long as 1:1; the unscaled horizon would have timed out most 1:3 trades and pushed Neutral straight back up, re-creating the imbalance the relabel removes. Now multiplied by slMult*tpMult, calibrated against a real measurement rather than assumed: the accidental 1:1 run resolved at horizon 12 with only 16.7% timeouts, so the swing median is the right scale at m*k=1. Verifiable, not just asserted: the prebuild now counts barriers that ended on the VERTICAL barrier and reports them as a share of Neutral. Neutral conflates "timed out" with "stopped out" and only the first indicts the horizon. Both builds compile 0 errors / 0 warnings. Forces a retrain - correcting TP_Mode re-keys the fingerprint (|TB:1:-101 -> |TB:1:3), which is right: no existing model was trained on the intended target. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:30:49 -04:00
logChancePrecPct = (chancePrecPct >= 0.0) ? (int)MathRound(chancePrecPct) : -1;
feat(ai): rank checkpoints on directional precision, not balanced accuracy Balanced accuracy is maximized by exactly the model this system must never deploy. Measured frontier at fixed signal strength, base rate 6.1%: tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0% tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3% tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5% It rises monotonically as the model calls MORE and is right LESS, because two of its three terms are directional recalls that a call-everything model drives to ~95%, while the Neutral term it sacrifices counts for only a third. The 2026-07-29 run landed exactly there: balanced 58-64% while calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base rate. Only the per-class recall floor stopped those deploying - a guard doing the job the objective should have been doing - and that same guard also rejected the genuinely useful sparse-but-precise checkpoints. Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how many were right. That is what a trading edge is. Two anti-degenerate floors bracket it, since precision alone is trivially maximized by calling almost nothing: coverage must reach a fraction of the true directional base rate (derived, not configured - it adapts to any symbol/timeframe/label rule), and precision must at least beat that base rate. Against the same frontier the deploy order inverts from tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first) to tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage) Balanced accuracy is kept in the log as a diagnostic and marked as such, so a run where the two disagree - the signature of an over-caller - is visible at a glance. MinRecall no longer decides what ships; it now only drives the diagnostic recall line and is a candidate for removal. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
//--- Deployability. Replaces the per-class recall floor as the gate the checkpoint
//--- selection and the plateau ladder's "is there anything safe to deploy" test read.
//--- MinRecall still drives the diagnostic recall line below, but no longer decides what
//--- ships - it is the input that produced the catch-22 where nothing ever qualified.
fix(autotune): MI scorer read an array nobody filled; add the permutation floor THE TUNER WAS A SILENT NO-OP. Every chart logged auto-tune complete - 17 candidate settings scored in ~139s, feature/label mutual information 0.0000 -> 0.0000 nats (no improvement) 0.0000 is not a weak result, it is a broken measurement: finite-sample MI is biased UPWARD, so even pure noise scores above zero. Cause: ScoreCurrentParamsByMI called BufferTempDataCompute(), which APPENDS the bar's features to TempData and never touches m_featureCache - only the caching wrapper BufferTempData() writes that array. It then read m_featureCache, which ReInitADIndicators had just invalidated. Every column came back constant, FeatureColumnMI returned 0 for all of them, and all 17 candidates tied at exactly zero. 139 s per chart to return the settings it started with. Now reads the values back out of TempData, where they actually land. And an exactly-zero best score is called out as a fault rather than reported as "no improvement", because that is what it is. ADDED: a PERMUTATION BASELINE, which is the diagnostic this project has been missing. MI's finite-sample bias is ~(bins-1)(classes-1)/(2n) nats - at these sample sizes the same order as any real edge in this domain - so a raw MI figure is uninterpretable on its own. Shuffling the labels destroys every genuine association while leaving sample size, binning and class proportions intact, so the score it produces IS this dataset's noise floor, measured rather than approximated. The log now reads feature/label information - X nats against a shuffled-label floor of Y and says outright whether the features carry usable information about the target. It needs no training, no topology and no convergence, so unlike every accuracy number in this codebase it cannot be confounded by an optimizer or an objective. If the score sits on the floor, no change of architecture can help - which is the question the last three days of zero-edge results have been circling. DEPLOY FLOOR: `dirPrecPct > chancePrecPct` passed anything above chance by any amount. At ~11,000 directional calls the standard error of the precision estimate is ~0.4pp, so that gate was accepting sub-one-sigma noise - the perceptron deployed at edge +0pp on 2026-08-01. Now requires EDGE_MIN_SIGMAS (2.0) standard errors above chance, computed from the actual call count, so the bar scales with the evidence instead of needing a hand-picked constant. Recorded with it, because it is why chance is the right reference at all: under a driftless random walk P(touch +k*ATR before -m*ATR) = m/(m+k), and the break-even win rate for a k:m reward:risk trade is ALSO m/(m+k). The label's own base rate IS the break-even rate, at every SL/TP setting. So "beats chance" and "is profitable" are the same test, and no choice of SL/TP can manufacture an edge - only prediction can. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:05:50 -04:00
//--- The margin is not arbitrary and not a knob: beating chance by any amount at all is a
//--- coin-flip result once the estimate's own sampling error is accounted for. With
//--- oosDirCalls directional calls at a chance rate p, the standard error of the measured
//--- precision is sqrt(p(1-p)/n) - about 0.4pp at the ~11,000 calls these runs produce - so
//--- `dirPrecPct > chancePrecPct` was passing models whose entire "edge" was under one
//--- sigma. Observed 2026-08-01: the perceptron deployed at edge +0pp.
//--- Requiring EDGE_MIN_SIGMAS standard errors instead scales the bar with the evidence:
//--- a sparse model needs a bigger measured edge to qualify than a dense one, which is
//--- exactly right, and no constant has to be re-tuned when coverage changes.
fix(labels): overlapping-label sample correction + horizon cap on the scale ladder Three defects, all surfaced by the 2026-08-17 SP500 H4 run that shipped stop 4.86 / target 9.71 (width 14.57*ATR, horizon 384). 1. EVERY STANDARD ERROR ASSUMED INDEPENDENT SAMPLES. Triple-barrier labels started one per bar overlap by the label's lifespan, so n calls are worth ~n/L independent observations (Lopez de Prado, AFML ch. 4 - sample uniqueness). All three sqrt(p(1-p)/n) sites divided by the RAW count. The tell: the operating point's null-of-the-maximum gate is family-wise and should fire on ~5% of eras under the null. Measured fire rates - PAI 47/73 (64%), ConvLSTM 9/24, LSTM 8/21 (38%), CONV 4/62 (6%). CONV, the only model whose margin distribution admits few bins, sat on the null; the rest cleared a bar that was too low by ~sqrt(L). PAI's deployed threshold consequently alternated between the ENDS of its own range era to era (0.10 -> 0.88 -> 0.86 -> 0.66; coverage 16% <-> 73%). TripleBarrierLabel now records when each label became KNOWABLE - the first winning touch, or both stops, or the timeout - and the prebuild accumulates the mean. EffectiveSampleSize() feeds the operating point, the member deploy gate and the ensemble vote gate. Conservative by construction (n/L is an upper bound on the damage); gates get harder, never easier. 2. THE SCALE LADDER RAN AWAY, again. Horizon scales as swingMedian*sl*tp, and since 4d8cb08 reachability is measured OVER that horizon - so a wider rung buys itself the time that makes it look reachable. Same target -> horizon -> reach -> target loop the excursion window is kept short to avoid; fixing the window confusion reopened it through the other door. It walked 128 -> 256 -> 384 bars and stopped at q90, the widest rung there is, with every rung reading 39-48% against a 20% floor. A floor nothing fails selects nothing. Rungs whose required horizon exceeds BARRIER_HORIZON_MAX are now rejected - the same rule ReportGeometryExpectancyScan already applied. It was printing the shipped pair as CLAMPED and disqualified ('h384!') two lines under the deriver that chose it: two subsystems, one geometry, opposite verdicts. 3. THE RUNG SNAP DESTROYED THE RATIO IT WAS COMPARING. Both legs snapped independently to the coarse first-passage grid, re-rating each candidate: q90 4.86/9.71 -> 5.00/10.00 (2.00), q85 4.07/8.14 -> 5.00/10.00 (IDENTICAL measurement), q75 3.07/6.13 -> 4.00/6.50 (1.63 - a nearer target). So the ladder compared win shares taken at ratios from 1.63 to 2.17 and read the differences as scale. It is why the reach column came out non-monotone in width (q75 48.5% above q90 42.9%). The stop now snaps to its nearest rung in log space and the target follows the ratio off it; the pair actually measured is returned and logged, so a collision reads as a collision. Also: LadderWinShare guarded against the conditional (fractal) geometry path, which fills n from m_fracLegCount while leaving idxList empty - a latent out-of-bounds on a currently-dead path. New log lines: mean label lifespan and effective n on the label-cache line, the required-vs-available horizon per rung, and the grid pair the reconciliation actually measured (its tolerance now scales with the grid skew instead of a flat 5pp). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 01:12:05 -04:00
//--- ON THE EFFECTIVE SAMPLE (2026-08-17). The comment above quotes "about 0.4pp at the
//--- ~11,000 calls these runs produce" - that figure assumed 11,000 INDEPENDENT calls.
//--- They are triple-barrier outcomes on consecutive bars, overlapping by the label's mean
//--- lifespan, so the real error is larger by ~sqrt(L) - at the 384-bar horizon this run
//--- shipped, an order of magnitude larger. Every "clears by N sigma" verdict in the
//--- project's history was computed against the optimistic figure. See
//--- EffectiveSampleSize() for the correction and the evidence that forced it.
fix(autotune): MI scorer read an array nobody filled; add the permutation floor THE TUNER WAS A SILENT NO-OP. Every chart logged auto-tune complete - 17 candidate settings scored in ~139s, feature/label mutual information 0.0000 -> 0.0000 nats (no improvement) 0.0000 is not a weak result, it is a broken measurement: finite-sample MI is biased UPWARD, so even pure noise scores above zero. Cause: ScoreCurrentParamsByMI called BufferTempDataCompute(), which APPENDS the bar's features to TempData and never touches m_featureCache - only the caching wrapper BufferTempData() writes that array. It then read m_featureCache, which ReInitADIndicators had just invalidated. Every column came back constant, FeatureColumnMI returned 0 for all of them, and all 17 candidates tied at exactly zero. 139 s per chart to return the settings it started with. Now reads the values back out of TempData, where they actually land. And an exactly-zero best score is called out as a fault rather than reported as "no improvement", because that is what it is. ADDED: a PERMUTATION BASELINE, which is the diagnostic this project has been missing. MI's finite-sample bias is ~(bins-1)(classes-1)/(2n) nats - at these sample sizes the same order as any real edge in this domain - so a raw MI figure is uninterpretable on its own. Shuffling the labels destroys every genuine association while leaving sample size, binning and class proportions intact, so the score it produces IS this dataset's noise floor, measured rather than approximated. The log now reads feature/label information - X nats against a shuffled-label floor of Y and says outright whether the features carry usable information about the target. It needs no training, no topology and no convergence, so unlike every accuracy number in this codebase it cannot be confounded by an optimizer or an objective. If the score sits on the floor, no change of architecture can help - which is the question the last three days of zero-edge results have been circling. DEPLOY FLOOR: `dirPrecPct > chancePrecPct` passed anything above chance by any amount. At ~11,000 directional calls the standard error of the precision estimate is ~0.4pp, so that gate was accepting sub-one-sigma noise - the perceptron deployed at edge +0pp on 2026-08-01. Now requires EDGE_MIN_SIGMAS (2.0) standard errors above chance, computed from the actual call count, so the bar scales with the evidence instead of needing a hand-picked constant. Recorded with it, because it is why chance is the right reference at all: under a driftless random walk P(touch +k*ATR before -m*ATR) = m/(m+k), and the break-even win rate for a k:m reward:risk trade is ALSO m/(m+k). The label's own base rate IS the break-even rate, at every SL/TP setting. So "beats chance" and "is profitable" are the same test, and no choice of SL/TP can manufacture an edge - only prediction can. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:05:50 -04:00
double chanceP = (chancePrecPct >= 0.0) ? chancePrecPct / 100.0 : 0.0;
double precSE = (oosDirCalls > 0 && chanceP > 0.0 && chanceP < 1.0)
fix(labels): overlapping-label sample correction + horizon cap on the scale ladder Three defects, all surfaced by the 2026-08-17 SP500 H4 run that shipped stop 4.86 / target 9.71 (width 14.57*ATR, horizon 384). 1. EVERY STANDARD ERROR ASSUMED INDEPENDENT SAMPLES. Triple-barrier labels started one per bar overlap by the label's lifespan, so n calls are worth ~n/L independent observations (Lopez de Prado, AFML ch. 4 - sample uniqueness). All three sqrt(p(1-p)/n) sites divided by the RAW count. The tell: the operating point's null-of-the-maximum gate is family-wise and should fire on ~5% of eras under the null. Measured fire rates - PAI 47/73 (64%), ConvLSTM 9/24, LSTM 8/21 (38%), CONV 4/62 (6%). CONV, the only model whose margin distribution admits few bins, sat on the null; the rest cleared a bar that was too low by ~sqrt(L). PAI's deployed threshold consequently alternated between the ENDS of its own range era to era (0.10 -> 0.88 -> 0.86 -> 0.66; coverage 16% <-> 73%). TripleBarrierLabel now records when each label became KNOWABLE - the first winning touch, or both stops, or the timeout - and the prebuild accumulates the mean. EffectiveSampleSize() feeds the operating point, the member deploy gate and the ensemble vote gate. Conservative by construction (n/L is an upper bound on the damage); gates get harder, never easier. 2. THE SCALE LADDER RAN AWAY, again. Horizon scales as swingMedian*sl*tp, and since 4d8cb08 reachability is measured OVER that horizon - so a wider rung buys itself the time that makes it look reachable. Same target -> horizon -> reach -> target loop the excursion window is kept short to avoid; fixing the window confusion reopened it through the other door. It walked 128 -> 256 -> 384 bars and stopped at q90, the widest rung there is, with every rung reading 39-48% against a 20% floor. A floor nothing fails selects nothing. Rungs whose required horizon exceeds BARRIER_HORIZON_MAX are now rejected - the same rule ReportGeometryExpectancyScan already applied. It was printing the shipped pair as CLAMPED and disqualified ('h384!') two lines under the deriver that chose it: two subsystems, one geometry, opposite verdicts. 3. THE RUNG SNAP DESTROYED THE RATIO IT WAS COMPARING. Both legs snapped independently to the coarse first-passage grid, re-rating each candidate: q90 4.86/9.71 -> 5.00/10.00 (2.00), q85 4.07/8.14 -> 5.00/10.00 (IDENTICAL measurement), q75 3.07/6.13 -> 4.00/6.50 (1.63 - a nearer target). So the ladder compared win shares taken at ratios from 1.63 to 2.17 and read the differences as scale. It is why the reach column came out non-monotone in width (q75 48.5% above q90 42.9%). The stop now snaps to its nearest rung in log space and the target follows the ratio off it; the pair actually measured is returned and logged, so a collision reads as a collision. Also: LadderWinShare guarded against the conditional (fractal) geometry path, which fills n from m_fracLegCount while leaving idxList empty - a latent out-of-bounds on a currently-dead path. New log lines: mean label lifespan and effective n on the label-cache line, the required-vs-available horizon per rung, and the grid pair the reconciliation actually measured (its tolerance now scales with the grid skew instead of a flat 5pp). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 01:12:05 -04:00
? 100.0 * MathSqrt(chanceP * (1.0 - chanceP)
/ EffectiveSampleSize((double)oosDirCalls)) : 0.0;
fix(autotune): MI scorer read an array nobody filled; add the permutation floor THE TUNER WAS A SILENT NO-OP. Every chart logged auto-tune complete - 17 candidate settings scored in ~139s, feature/label mutual information 0.0000 -> 0.0000 nats (no improvement) 0.0000 is not a weak result, it is a broken measurement: finite-sample MI is biased UPWARD, so even pure noise scores above zero. Cause: ScoreCurrentParamsByMI called BufferTempDataCompute(), which APPENDS the bar's features to TempData and never touches m_featureCache - only the caching wrapper BufferTempData() writes that array. It then read m_featureCache, which ReInitADIndicators had just invalidated. Every column came back constant, FeatureColumnMI returned 0 for all of them, and all 17 candidates tied at exactly zero. 139 s per chart to return the settings it started with. Now reads the values back out of TempData, where they actually land. And an exactly-zero best score is called out as a fault rather than reported as "no improvement", because that is what it is. ADDED: a PERMUTATION BASELINE, which is the diagnostic this project has been missing. MI's finite-sample bias is ~(bins-1)(classes-1)/(2n) nats - at these sample sizes the same order as any real edge in this domain - so a raw MI figure is uninterpretable on its own. Shuffling the labels destroys every genuine association while leaving sample size, binning and class proportions intact, so the score it produces IS this dataset's noise floor, measured rather than approximated. The log now reads feature/label information - X nats against a shuffled-label floor of Y and says outright whether the features carry usable information about the target. It needs no training, no topology and no convergence, so unlike every accuracy number in this codebase it cannot be confounded by an optimizer or an objective. If the score sits on the floor, no change of architecture can help - which is the question the last three days of zero-edge results have been circling. DEPLOY FLOOR: `dirPrecPct > chancePrecPct` passed anything above chance by any amount. At ~11,000 directional calls the standard error of the precision estimate is ~0.4pp, so that gate was accepting sub-one-sigma noise - the perceptron deployed at edge +0pp on 2026-08-01. Now requires EDGE_MIN_SIGMAS (2.0) standard errors above chance, computed from the actual call count, so the bar scales with the evidence instead of needing a hand-picked constant. Recorded with it, because it is why chance is the right reference at all: under a driftless random walk P(touch +k*ATR before -m*ATR) = m/(m+k), and the break-even win rate for a k:m reward:risk trade is ALSO m/(m+k). The label's own base rate IS the break-even rate, at every SL/TP setting. So "beats chance" and "is profitable" are the same test, and no choice of SL/TP can manufacture an edge - only prediction can. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:05:50 -04:00
double edgeFloorPct = chancePrecPct + EDGE_MIN_SIGMAS * precSE;
feat(measurement): fix zero-skill denominator, publish the deploy bar, measure lifespan per rung, add a MEASURE scale objective The last run could not have demonstrated an edge either way, and nothing in the log said so. Four changes so it does. 1. THE ZERO-SKILL LINE DIVIDED BY THE WRONG DENOMINATOR. m_oosWinLongTotal resets every era; m_oosSamples only resets on a full model reset. So 'always-long %' decayed as ~1/era: a run whose true rate is 37% printed 1.2% at era 33 and 0.0% at era 2219. This is the SAME bug already found and fixed for logBuyPredPct thirty lines above ('era-15 Buy:2% that was really ~30%'), left in the one line whose whole job is to be the reference every other number is read against. Correct at era 1, wrong everywhere after - including the '62% zero-skill' figure in the 2026-08-16 notes. Now per-era, and always-short is finally readable. 2. THE DEPLOY GATE STATES ITS OWN BAR. 'edge -1pp' era after era cannot separate 'short by a hair' from 'short by an amount no strategy could cover'. The era line now prints the required win rate, the SE, the effective n and the lifespan it was deflated by; above 100% it says UNREACHABLE. At 4,738 OOS bars and L=75.6 there are ~63 independent observations, putting the bar near 66% at typical coverage. 3. LIFESPAN MEASURED PER RUNG. The first-passage cache already stores touch ages at every ladder level, so each candidate geometry's resolution time is readable without training on it - L-vs-width becomes a measurement across the whole ladder in ONE run rather than a second chart. Each rung reports L, n_eff, min provable edge and min provable EV. 4. SCALE OBJECTIVE IS PHASE-AWARE, defaulting to MEASURE. Width and detectability are opposed: labels overlap by L, L grows like m*k = width^2 at fixed ratio, so min provable EV ~ width^2 while the cost saving from width is only linear. Doubling width quadruples the smallest EV you can prove. DEPLOY (widest that clears reachability) is right once an edge is known; MEASURE (narrowest that keeps round-trip spread under BARRIER_MAX_COST_FRACTION_PCT) is right while it still has to be shown. The direction does not depend on the exponent, and item 3 makes the exponent checkable. Fixed in review: m_lastRungLifespan is cleared on every LadderWinShare entry or a rejected rung reports the previous rung's lifespan as its own; per-rung detectability is labelled IS-sample based (the deriver may not see the holdout), so absolute figures are optimistic while the ranking is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:37:05 -04:00
//--- PUBLISHED so the era line can state the bar instead of leaving it implicit. A gate
//--- that is arithmetically unreachable must SAY so: with a 4,738-bar OOS window and a
//--- 75.6-bar mean label lifespan there are only ~63 independent observations in it, and
//--- at 18% coverage that is ~11 - which puts the required win rate near 66% against a 37%
//--- chance rate. Nothing will ever clear that, and until this line printed it the
//--- symptom was indistinguishable from "the models are close but not quite".
m_lastEdgeFloorPct = edgeFloorPct;
m_lastPrecSE = precSE;
m_lastEffN = EffectiveSampleSize((double)oosDirCalls);
feat(gate): cross-instrument pooled certification The deploy bottleneck is CERTIFICATION, not training. A 4,738-bar OOS window at L=75.6 holds ~63 independent observations; certifying a 3pp edge at 2 sigma needs ~1,036. More bars of the same symbol barely help - they overlap. Other symbols do not. WHAT POOLS. Not win rates: symbols have different derived geometries, different break-evens and different drifts, so averaging raw rates across them is meaningless. What pools is each symbol's EXCESS OVER ITS OWN CHANCE RATE, combined by inverse-variance weighting (fixed-effects meta-analysis). Each symbol keeps its own model, geometry and chance rate; only the evidence is combined. THE CORRELATION PROBLEM, bracketed rather than assumed away. SP500 and NAS100 are ~0.9 correlated and pooling them as independent inflates the evidence. Nothing here can measure that without sharing return series, so instead of guessing a correction the gate reports both ends: SE_INDEP = sqrt(1/SUM(1/var_i)) all members independent SE_CORR = SUM(w_i * sqrt(var_i)) all members perfectly correlated The truth is always between. THE GATE USES SE_CORR, so a pass cannot be an artifact of correlated instruments - that bound already assumes the worst. The ratio is logged as the diversification credit the gate declines to claim, so the cost of that conservatism is visible instead of hidden. SCOPE, deliberately limited: the pooled result is REPORTED, never folded into tradeableOK. The local gate certifies the model that actually trades this symbol; the pool answers the different question of whether the strategy has an edge at all. Letting a cross-symbol result license a local deploy would ship a model that never cleared its own bar - so it cannot. Mechanics: one file per instrument (no concurrent-write path to get wrong), every FileOpen carrying FILE_SHARE_READ|FILE_SHARE_WRITE, records skipped rather than reinterpreted on a version mismatch, 12h staleness cutoff so a stopped chart cannot vote, and pooling refused below 3 instruments. Poolability requires matching timeframe and ratio; differing SYMBOL is the entire point. Publishing is unconditional - a pool that only hears from winners is a selection effect, not a meta-analysis. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:45:37 -04:00
//--- CONTRIBUTE THIS ERA'S EVIDENCE TO THE CROSS-INSTRUMENT POOL, then read the pool
//--- back. Publishing unconditionally - not only when the local gate passes - because a
//--- symbol that is short of its own bar is still evidence about whether the STRATEGY
//--- has an edge, and a pool that only hears from winners is a selection effect, not a
//--- meta-analysis. See PooledGate.mqh.
if(coverageMeasurable && dirPrecPct >= 0.0 && chancePrecPct > 0.0)
{
PublishPoolRecord(chancePrecPct, dirPrecPct, m_lastEffN);
m_lastPoolPasses = PooledGatePasses(m_lastPoolReport);
}
fix: the Adam second moment was never Adam - all four tiers Root cause of the B=32 regression, and it predates F4 entirely. Every Adam kernel stored v already square-rooted and then fed that stored value back in as if it were the variance: v_new = sqrt(b2 * v_old + (1 - b2) * g^2) That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below unit scale, so the denominator stops tracking the gradient and Adam degrades into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll (batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where a scale-invariant optimizer gives the same distance for both. After the fix all six magnitudes read 1.199 and v tracks |g| exactly. It hit conv/LSTM specifically because they sit behind a batch-norm with running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep in the degraded regime - while the dense stack near the loss stayed in the working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/ 0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already squared v back for gamma/beta and its comment named the kernels as wrong, which is exactly why gamma/beta kept training while the stages behind froze. Persisted .nnw needs no migration - v keeps its std-dev meaning. Also, the two ways F4 exposed it, both mine: - No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods (Krizhevsky 2014; Granziol et al. 2022), applied once in InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD. - Plateau patience denominated in eras, so raising B made the ladder 32x more impatient in its only unit. PAI converged at era 41 on ~49k updates where the same config had been finding new bests at era 1028. TrainPlateauPatienceEras() stretches it by the same sqrt(B). TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23 eras per stage, not 8 -> 45). Both helpers are identities at B=1. Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into the ranking key rather than checked at deploy time so a one-sided era cannot become best-so-far in the first place. Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded CAppDialog teardown that sat ahead of it - the same ordering inversion the rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit (vs ~1.1 s for the three that finished) having reached none of its cleanup, so its arrows stayed on the chart. Steps are now timed in the log. PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now walks every object type and reports the object counts when both are zero. Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt. FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
//--- BOTH sides must still be alive - see DEPLOY_MIN_SIDE_RECALL_PCT. A negative recall
//--- means "not measurable this era" (no true bars of that class in the OOS window), and
//--- that must not be read as a dead side, so it passes.
bool bothSidesLive = (buyRecallPct < 0 || buyRecallPct >= DEPLOY_MIN_SIDE_RECALL_PCT) &&
(sellRecallPct < 0 || sellRecallPct >= DEPLOY_MIN_SIDE_RECALL_PCT);
//--- Folded into tradeableOK rather than checked only at deploy time, deliberately: this
//--- flag is also the lexicographic ranking key (isBetterEra) and the eta-decay trigger,
//--- so a one-sided era must not be allowed to become the best-so-far in the first place.
//--- Checking it only at the deploy gate would let the ladder spend its whole patience
//--- budget ranking one-sided eras against each other and then refuse to ship the winner.
bool tradeableOK = coverageMeasurable && dirPrecPct >= 0.0 && bothSidesLive &&
fix(autotune): MI scorer read an array nobody filled; add the permutation floor THE TUNER WAS A SILENT NO-OP. Every chart logged auto-tune complete - 17 candidate settings scored in ~139s, feature/label mutual information 0.0000 -> 0.0000 nats (no improvement) 0.0000 is not a weak result, it is a broken measurement: finite-sample MI is biased UPWARD, so even pure noise scores above zero. Cause: ScoreCurrentParamsByMI called BufferTempDataCompute(), which APPENDS the bar's features to TempData and never touches m_featureCache - only the caching wrapper BufferTempData() writes that array. It then read m_featureCache, which ReInitADIndicators had just invalidated. Every column came back constant, FeatureColumnMI returned 0 for all of them, and all 17 candidates tied at exactly zero. 139 s per chart to return the settings it started with. Now reads the values back out of TempData, where they actually land. And an exactly-zero best score is called out as a fault rather than reported as "no improvement", because that is what it is. ADDED: a PERMUTATION BASELINE, which is the diagnostic this project has been missing. MI's finite-sample bias is ~(bins-1)(classes-1)/(2n) nats - at these sample sizes the same order as any real edge in this domain - so a raw MI figure is uninterpretable on its own. Shuffling the labels destroys every genuine association while leaving sample size, binning and class proportions intact, so the score it produces IS this dataset's noise floor, measured rather than approximated. The log now reads feature/label information - X nats against a shuffled-label floor of Y and says outright whether the features carry usable information about the target. It needs no training, no topology and no convergence, so unlike every accuracy number in this codebase it cannot be confounded by an optimizer or an objective. If the score sits on the floor, no change of architecture can help - which is the question the last three days of zero-edge results have been circling. DEPLOY FLOOR: `dirPrecPct > chancePrecPct` passed anything above chance by any amount. At ~11,000 directional calls the standard error of the precision estimate is ~0.4pp, so that gate was accepting sub-one-sigma noise - the perceptron deployed at edge +0pp on 2026-08-01. Now requires EDGE_MIN_SIGMAS (2.0) standard errors above chance, computed from the actual call count, so the bar scales with the evidence instead of needing a hand-picked constant. Recorded with it, because it is why chance is the right reference at all: under a driftless random walk P(touch +k*ATR before -m*ATR) = m/(m+k), and the break-even win rate for a k:m reward:risk trade is ALSO m/(m+k). The label's own base rate IS the break-even rate, at every SL/TP setting. So "beats chance" and "is profitable" are the same test, and no choice of SL/TP can manufacture an edge - only prediction can. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:05:50 -04:00
coveragePct >= minCoveragePct && dirPrecPct > edgeFloorPct;
fix(ai): discount selection precision by coverage shortfall The 2026-07-30 run caught a bug in the precision-led selection metric within 8 eras. HYBRID made exactly ONE directional call in era 7, got it right, scored 100% precision, and locked that in as best-ever. Nothing can beat 100%, so the checkpoint froze on a single sample and the run could only burn to the era cap deploying it. The coverage floor already existed and already blocked that era from being DEPLOYABLE - but the ranking ignored coverage entirely whenever no era had qualified yet, which is precisely the phase where the ranking is the only thing steering the run. Precision is now discounted by coverage/floor, capped at 1.0. Continuous rather than a threshold: an era at half the floor scores half its precision, so coverage and precision both improve rank and neither can be traded away. Above the floor the credit saturates, so ranking among genuinely deployable eras is unchanged pure precision. Also: the startup config line printed "tau 1.00" while every chart was actually running the capped 0.35 - the effective value depends on the measured class priors and is not knowable at init. Now reads "1.00 requested"; ApplyLogitAdjustment still logs the real figure. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:40:36 -04:00
//--- Ranking key: precision, DISCOUNTED by how far short of the coverage floor the era
//--- fell. Raw precision was wrong here and the 2026-07-30 run caught it within 8 eras -
//--- HYBRID made exactly ONE directional call, got it right, scored 100%, and locked that
//--- in as best-ever. Nothing can beat 100%, so the checkpoint was frozen on a single
//--- sample and the run could only burn to the era cap. The coverage floor was already
//--- computed and already blocked that era from being DEPLOYABLE, but the ranking ignored
//--- it whenever no era had qualified yet - which is exactly the phase this matters in.
//--- Discounting rather than thresholding keeps the ordering continuous: an era at half
//--- the floor scores half its precision, so more coverage and better precision both
//--- improve rank and neither can be traded away entirely. Above the floor the credit
//--- saturates at 1.0, so ranking among genuinely deployable eras stays pure precision.
double coverageCredit = 1.0;
if(minCoveragePct > 0.0 && coveragePct >= 0.0)
coverageCredit = MathMin(1.0, coveragePct / minCoveragePct);
double selectionScore = (dirPrecPct >= 0.0) ? dirPrecPct * coverageCredit : 0.0;
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- THE S2 REPORT (Meta_Labeling_Design.md): the meta head's era verdict in the
//--- design's own terms - coverage x (win rate - break-even) against the no-skill null.
//--- Two references on purpose: chancePrecPct (the base win rate + its SE) answers
//--- "does the head KNOW anything", the geometric break-even answers "would trading its
//--- calls MAKE anything" - a head can clear the first and still sit under the second
//--- when the candidate stream itself is unprofitable (the honest-floor case).
if(IsMetaTarget() && coverageMeasurable && oosEraBars > 0)
{
double mSl, mTp;
BarrierMultiples(mSl, mTp);
double mBePct = (mSl + mTp > 0.0) ? 100.0 * mSl / (mSl + mTp) : 50.0;
double mScore = (dirPrecPct >= 0.0 && coveragePct >= 0.0)
? coveragePct * (dirPrecPct - mBePct) / 100.0 : 0.0;
PrintFormat("%s: META era %d - %d candidates OOS, base win %.1f%% | trades %d (%.1f%%"
" coverage) at %.1f%% win vs %.1f%% break-even -> cov x (p-BE) = %+.2f |"
" skill vs base %+.1fpp (needs > %+.1fpp at %d sigma) %s",
ID, (int)m_eraCount, oosEraBars, chancePrecPct, oosDirCalls, coveragePct,
dirPrecPct, mBePct, mScore,
dirPrecPct - chancePrecPct, EDGE_MIN_SIGMAS * precSE, (int)EDGE_MIN_SIGMAS,
tradeableOK ? "| DEPLOYABLE this era" : "");
feat: per-family x per-side OOS breakdown in the META era report 350-era S2 verdict on SP500 H1: the meta head carries REAL ranking skill (+1.0-1.3pp mean over base, 101/350 eras clear their own 2-sigma bar, traded subset wins 66.1% at <30% coverage vs 64.5% base) but 0/350 eras produced a positive cov x (p - BE): the candidate stream sits 3pp under the derived geometry's 67.5% break-even and ~2.6pp of recovered skill cannot bridge it. Skill plateaued by mid-run (1.28pp -> 1.05pp), so more eras only buy multiplicity, and the deploy gate correctly shipped nothing. The aggregate can hide a deployable subset (one family/side clearing BE blended with junk), so the META era line now decomposes the SAME traded population into MA/RSI/MACD/Ichimoku x LONG/SHORT cells, each as traded/candidates base->traded win rate. 32 cells is a best-of-N search by construction - any candidate cell faces the family-wise rule before belief. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 09:03:40 -04:00
//--- The decomposition the aggregate can hide (see the member declaration): each
//--- cell reads "traded/candidates base->traded win rate". A cell whose traded win
//--- clears mBePct on real volume is a deployable SUBSET even when the blend is not;
//--- judge it against the family-wise rule before believing it (32 cells is a
//--- best-of-N search by construction).
string famLine = "";
for(int mf = 0; mf < 4; mf++)
{
double fb = (m_metaFamCand[mf] > 0) ? 100.0 * m_metaFamWins[mf] / m_metaFamCand[mf] : 0.0;
double fw = (m_metaFamFired[mf] > 0) ? 100.0 * m_metaFamFiredWins[mf] / m_metaFamFired[mf] : 0.0;
famLine += StringFormat("%s %d/%d %.1f->%.1f%% ", MetaFamilyName(mf),
m_metaFamFired[mf], m_metaFamCand[mf], fb, fw);
}
double lb = (m_metaSideCand[0] > 0) ? 100.0 * m_metaSideWins[0] / m_metaSideCand[0] : 0.0;
double lw = (m_metaSideFired[0] > 0) ? 100.0 * m_metaSideFiredWins[0] / m_metaSideFired[0] : 0.0;
double sb = (m_metaSideCand[1] > 0) ? 100.0 * m_metaSideWins[1] / m_metaSideCand[1] : 0.0;
double sw = (m_metaSideFired[1] > 0) ? 100.0 * m_metaSideFiredWins[1] / m_metaSideFired[1] : 0.0;
PrintFormat("%s: META breakdown (traded/cands base->traded win, BE %.1f%%): %s|"
" LONG %d/%d %.1f->%.1f%% SHORT %d/%d %.1f->%.1f%%",
ID, mBePct, famLine,
m_metaSideFired[0], m_metaSideCand[0], lb, lw,
m_metaSideFired[1], m_metaSideCand[1], sb, sw);
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
}
fix: the recall gate was unsatisfiable and the LR decay was a spiral Both made the run structurally unable to succeed, independently of any signal in the data. Found by reading the 13:01 log. RECALL GATE. m_objectiveMet required Buy, Sell AND Neutral OOS recall each >= 40%. First-touch resolution (ce52654) collapsed Neutral from the ~94% majority it was under exact-pivot labels to a same-bar-tie residue - 250 of 38,261 bars, 0.65% - so the floor was asking the model to identify 40% of coin-flip ties before it could converge. Measured: CONV, LSTM and HYBRID all logged "Neutral:0% (need >=40% each)" on every era. No model could ever satisfy it; every run was destined for the plateau ladder or the era cap. Only the DIRECTIONAL floors are load-bearing for the anti-collapse job the gate exists to do: an all-Neutral model shows Buy and Sell recall at 0% and is blocked by them. Neutral's own floor guarded the mirror bias (over-calling Buy/Sell at Neutral's expense), which was real at 94% prevalence and is not at 0.65% - there, almost never calling Neutral is correct rather than biased. Prevalence-guarded rather than hardcoded off, so it returns by itself if a future label rule makes Neutral substantial again. Deliberately NOT extended to Buy/Sell: exempting a thin directional class reopens the era-44-46 hole, which directionalRecallMeasured only half-covers - it checks those classes were MEASURED, not that they passed. ETA DECAY. A regressing era restored the checkpoint, reset the optimizer and cut eta - all on the FIRST regression. The next era then started from an identical state with a smaller step, regressed again, and got the same treatment. The loop is self-sustaining and cannot discover anything, because rolling the weights back is exactly what removes the exploration that would end it. Measured on PAI: eras 2-11 every one a regression against era 1, eta 0.000594 -> 0.000024, dW/W 0.000%/0.000% from era 2 onward. Ten eras, ~45s each, reproducing era 1 exactly and unable to do anything else. Now requires ETA_DECAY_PATIENCE_ERAS consecutive regressions - the standard ReduceLROnPlateau formulation. A single bad era is noise, and an improving era clears the counter so alternating runs never accumulate into a decay. Build tag -> gate-patience-v3. It had not moved in six commits, which is why the running binary could not be identified from its own log. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:28:58 -04:00
//--- NEUTRAL CANNOT BLOCK WHEN IT IS TOO RARE TO LEARN. The floor exists to stop a
//--- one-class collapse, and for that only the DIRECTIONAL floors are load-bearing: a
//--- model that called everything Neutral would show Buy and Sell recall at 0% and be
//--- blocked by them. Neutral's own floor was protecting against the mirror bias
//--- (over-calling Buy/Sell at Neutral's expense) - which was a real risk when Neutral
//--- was the ~94% majority under exact-pivot labels, and stopped being one when
//--- first-touch resolution reduced it to a 0.65% same-bar-tie residue. At that
//--- prevalence, almost never calling Neutral is CORRECT rather than biased, so the
//--- floor was demanding the model be wrong in a specific way before it could converge.
//--- Prevalence-guarded rather than hardcoded off, so it comes back by itself if a
//--- future label rule makes Neutral substantial again.
//--- Deliberately NOT extended to Buy/Sell: exempting a thin directional class would
//--- reopen the era-44-46 hole (converging on a window with no directional bars to
//--- disprove the model), which directionalRecallMeasured below only half-covers - it
//--- checks those classes were MEASURED, not that they passed.
int neutralGatePct = neutralRecallPct;
if(oosEraBars > 0 &&
(100.0 * m_oosNeutralTotal / oosEraBars) < MIN_GATE_CLASS_SHARE_PCT)
neutralGatePct = -1;
feat(gates): derive the recall floor instead of configuring it, publish what the configuration can PROVE, and stop an arbitrary member driving live exits Three changes, all from the same principle: measure what is there before aiming at it, and never certify a number you do not trade. 1. THE RECALL FLOOR IS DERIVED, AND IT MOVES BELOW CHANCE. MinRecall=40 was a constant doing a statistical job. Its reference point is the 33.3% recall a zero-skill 3-class model gets on EVERY class, and against that the constant was accidentally calibrated for exactly one sample size: on USDJPY CONV (n_eff 195) 40% is chance + 2.0 SE; on SP500 PAI (n_eff 42) the same 40% is chance + 0.9 SE. One chart was being held to a bar twice as strict as the other, for no reason anyone chose. CollapseRecallFloorPct() computes it per class from that class's own effective sample - EffectiveSampleSize(), so the overlap deflation the rest of the gates use applies here too - as chance - EDGE_MIN_SIGMAS x SE. 26.5% at n_eff 195, 18.8% at n_eff 42. BELOW chance, deliberately, and this is the substantive change rather than the arithmetic. This gate's only job is refusing to call a COLLAPSED model converged. It is not a quality bar; the deploy gate is the quality bar and it is already rigorous (chance + 2 SE on the deflated sample, Sidak over candidate eras, then the cross-instrument pooled certificate). A convergence gate that ALSO demands provably-above-chance recall on all three classes double-counts that job, and it has failed that way twice here: MinRecall=60 blocked every SP500 H1 run in 2026-07, and the 40 that replaced it made Neutral structurally unreachable once first-touch resolution cut Neutral to a 0.65% residue. A floor nothing can reach does not make a funded account safer, it stops the run converging at all. Testing significantly BELOW chance instead catches what a fixed 40 was actually catching - a model that has stopped emitting a class - and cannot become unreachable by construction. It also fixes the direction the old constant scaled: it now widens on a thin OOS window, where low recall genuinely cannot be told from noise, and tightens on a rich one. Today's SP500 PAI (Buy 51 / Sell 18 / Neutral 30) is still correctly blocked on Sell. This also resolves a standing contradiction the code half-admitted at the isBetterEra comment: selection ranks on coverage-weighted PRECISION while convergence gated on RECALL, so a sparse high-precision abstainer - precisely the model that could clear the deploy bar - was blocked by the floor. The era line now PRINTS the derived floor. Anyone comparing these recalls against a remembered "40" is reading the wrong bar. 2. DETECTABILITY: WHAT THIS CONFIGURATION COULD PROVE, BEFORE IT TRAINS. The DEPLOY BAR line states the bar. It never said what reaching it would take, and that is the actionable direction. ReportDetectability() inverts the same identity - the gate passes when edge >= z x sqrt(p(1-p)/n_eff), so certifying an edge d needs n_eff >= z^2 p(1-p)/d^2 independent calls, hence L times as many raw ones - and prints a +2 / +5 / +10pp ladder as required independent calls, raw calls, and share of the OOS window, marking any rung that needs more than the window holds IMPOSSIBLE. Every term is a property of the CONFIGURATION - geometry via break-even, horizon via mean label lifespan, window via oosCutoff - so no amount of training moves any of them. It fires once, at the first healthy sweep, beside ReportFeatureHealth, for the same reason: that is the first moment the bar grid, the measured geometry and the lifespan are real numbers rather than defaults. It gates nothing. This is the EdgeFinder discipline applied to our own gate: establish what the market and the measurement design have to offer, then point the net at it - rather than spending a thousand eras chasing something this OOS window could never certify. 3. AN ARBITRARY MEMBER WAS DRIVING LIVE EXITS AND TRAILING (user-identified). Every ensemble member ran g_LiveAISignedConfidence = SignedAIConfidence(); unconditionally, every tick. Last writer wins. Its consumers are the AI early-exit route (CExpertSignalCustom::LiveSignedConfidence) and TrailingIntelligent - so on a four-model chart an LSTM entry could be closed, and its stop moved, on the Perceptron's opinion alone, decided by scheduling order. Not the vote, not a weighted blend. Now the mean across registered members, matching how the ensemble actually trades: the open decision is the weighted-average vote, and an abstaining member contributes 0 and dilutes exactly as it does there. Members still training read 0, so a half-trained ensemble reads WEAKER rather than louder - the safe direction for an exit trigger. Deployed and paused members are included, which is the opposite of the era barrier's exemption rule and correct for the opposite reason: that one asks who must be waited for, this asks who has an opinion. Latent today and staying that way for now by choice - Min_Vote_Close ships Disabled (101, unreachable on both scales it drives) and TrailingStrategy is off, so live exits are SL/TP only and the certified hold-to-barrier win rate is what actually gets traded. Fixed now precisely because the plan is to enable vote exits once the models are accurate, at which point a scheduling-order exit would be both harmful and very hard to see. STILL OPEN, and needs a decision before vote exits go on: the member gate and the ensemble vote gate both grade hold-to-barrier, so enabling vote exits makes the certified number stop describing the traded one. Warrior_EA.mq5 currently argues barrier models may keep vote exits because "their label IS the vote's own horizon" - that does not hold, since a vote flip at bar 5 of a 64-bar horizon is not the target-before-stop outcome the gate measured. Either grade the OOS call on the real exit rule (first of SL / TP / vote-flip / horizon) through the fill engine, or set HoldToBarrier for ensemble members so the policy cannot drift from the certificate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:15:08 -04:00
//--- DERIVED, per class, from that class's own effective sample - see
//--- CollapseRecallFloorPct() for why it sits BELOW chance rather than above it, and for
//--- the two occasions a fixed constant here made convergence structurally impossible.
double buyFloor = CollapseRecallFloorPct(m_oosBuyTotal);
double sellFloor = CollapseRecallFloorPct(m_oosSellTotal);
double neutralFloor = CollapseRecallFloorPct(m_oosNeutralTotal);
m_lastRecallFloorPct = (buyFloor + sellFloor + neutralFloor) / 3.0;
bool directionalRecallOK = (buyRecallPct < 0 || buyRecallPct >= buyFloor) &&
(sellRecallPct < 0 || sellRecallPct >= sellFloor) &&
(neutralGatePct < 0 || neutralGatePct >= neutralFloor);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// Balanced accuracy (macro-recall): the mean of the three per-class recalls - the metric
// the checkpoint SELECTION ranks on (see m_bestBalancedOos). Unlike blended accuracy it
// weights Buy, Sell and Neutral equally, so it can't be inflated by the ~96%-Neutral base
// rate. Computed from the same raw per-era recalls the floor uses (not smoothed - the
// whole recall-driven side of this block is per-era-raw by design). When a directional
// class is thin/unmeasured this era (recall -1), balanced accuracy isn't meaningful, so
// fall back to the blended dOosForecast for ranking that era (prior behavior) rather than
// averaging a partial set - the recall floor + directionalRecallMeasured still guard the
// actual convergence decision separately.
double balancedOosEra = (buyRecallPct >= 0 && sellRecallPct >= 0 && neutralRecallPct >= 0)
? (buyRecallPct + sellRecallPct + neutralRecallPct) / 3.0
: dOosForecast;
logBalancedAccPct = (buyRecallPct >= 0 && sellRecallPct >= 0 && neutralRecallPct >= 0)
? (int)MathRound(balancedOosEra) : -1;
// A real (non-thin-sample, i.e. not the -1 "n/a" sentinel) 0% recall on any class means
// the model never once got that class right this era - a majority-class collapse
// (predict-everything-Neutral, or symmetrically a Buy/Sell-only collapse), not progress
// toward separating classes. Before any era has ever passed the recall floor,
// isBetterEra's fallback below is a pure blended-accuracy tiebreak, and blended accuracy
// is trivially maximized by collapsing to the majority class. Observed in practice
// (2026-07-19, SP500 H4): once a run landed on a 0%/0%/100% Buy/Sell/Neutral era, its
// accuracy kept creeping upward for 124 STRAIGHT eras purely from sharpening the
// Neutral-vs-everything boundary - each tick registered as a "new best", re-anchoring the
// checkpoint AND bumping eta back toward its ceiling (the recovery bump below), actively
// rewarding the collapse instead of remaining neutral to it. Excluding these eras from
// isBetterEra denies them that anchor/reward without touching the restore/decay branch
// below, which stays exactly as gated on m_bestPassedRecall as before - see that block's
// own comment for why loosening THAT part pre-pass caused a worse failure historically.
feat(ai): rank checkpoints on directional precision, not balanced accuracy Balanced accuracy is maximized by exactly the model this system must never deploy. Measured frontier at fixed signal strength, base rate 6.1%: tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0% tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3% tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5% It rises monotonically as the model calls MORE and is right LESS, because two of its three terms are directional recalls that a call-everything model drives to ~95%, while the Neutral term it sacrifices counts for only a third. The 2026-07-29 run landed exactly there: balanced 58-64% while calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base rate. Only the per-class recall floor stopped those deploying - a guard doing the job the objective should have been doing - and that same guard also rejected the genuinely useful sparse-but-precise checkpoints. Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how many were right. That is what a trading edge is. Two anti-degenerate floors bracket it, since precision alone is trivially maximized by calling almost nothing: coverage must reach a fraction of the true directional base rate (derived, not configured - it adapts to any symbol/timeframe/label rule), and precision must at least beat that base rate. Against the same frontier the deploy order inverts from tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first) to tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage) Balanced accuracy is kept in the log as a diagnostic and marked as such, so a run where the two disagree - the signature of an over-caller - is visible at a glance. MinRecall no longer decides what ships; it now only drives the diagnostic recall line and is a candidate for removal. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
//--- Under precision ranking the degenerate era is the one that called NOTHING
//--- directional (precision undefined, nothing to trade), not one whose per-class
//--- recall touched zero - a sparse high-precision model legitimately has low recall.
bool isFullyCollapsedEra = (oosDirCalls <= 0);
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
//--- N for the family-wise deployment gate. Counted here, next to the exclusion it mirrors:
//--- an era that called nothing directional can never become the best (isBetterEra excludes
//--- it), so counting it would inflate N and make the gate stricter than the search that
//--- actually happened. Every era that COULD have won is counted, whether it did or not -
//--- that is precisely the set the maximum was taken over.
if(coverageMeasurable && !isFullyCollapsedEra)
m_deployCandidateEras++;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// Lexicographic "better than the best-so-far" ordering: passing the directional recall
// floor always outranks not passing it, regardless of blended dOosForecast; only WITHIN
// the same pass/fail category does blended accuracy break the tie. Without this, an era
// that traded a few "safe" Neutral calls for genuinely useful (recall-improving) Buy/Sell
// calls would look like a regression in blended-accuracy-only terms and get its
// checkpoint skipped / learning rate cut - fighting directly against the network learning
// to call Buy/Sell at all, since Neutral is the large majority class (~80%+ of labels) and
// a model that just calls everything Neutral already scores well on blended accuracy
// alone. isWorseEra mirrors the same ordering for the eta-decay-on-regression trigger.
// (directionalRecallOK implies !isFullyCollapsedEra already, since the floor is always
// >0%, so the first clause below needs no extra guard - only the pre-pass accuracy-only
// tiebreak in the second clause does.)
// Within the same recall-pass category the tie now breaks on BALANCED accuracy, not the
// Neutral-dominated blended dOosForecast - see m_bestBalancedOos. This is what deploys the
// most class-balanced era instead of the most Neutral-leaning one, and it also strengthens
// the pre-pass phase: a Neutral-only era scores (0+0+N)/3 in balanced terms (low) rather
// than the ~80% it scores in blended terms, so it can no longer re-anchor the checkpoint.
feat(ai): rank checkpoints on directional precision, not balanced accuracy Balanced accuracy is maximized by exactly the model this system must never deploy. Measured frontier at fixed signal strength, base rate 6.1%: tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0% tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3% tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5% It rises monotonically as the model calls MORE and is right LESS, because two of its three terms are directional recalls that a call-everything model drives to ~95%, while the Neutral term it sacrifices counts for only a third. The 2026-07-29 run landed exactly there: balanced 58-64% while calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base rate. Only the per-class recall floor stopped those deploying - a guard doing the job the objective should have been doing - and that same guard also rejected the genuinely useful sparse-but-precise checkpoints. Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how many were right. That is what a trading edge is. Two anti-degenerate floors bracket it, since precision alone is trivially maximized by calling almost nothing: coverage must reach a fraction of the true directional base rate (derived, not configured - it adapts to any symbol/timeframe/label rule), and precision must at least beat that base rate. Against the same frontier the deploy order inverts from tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first) to tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage) Balanced accuracy is kept in the log as a diagnostic and marked as such, so a run where the two disagree - the signature of an over-caller - is visible at a glance. MinRecall no longer decides what ships; it now only drives the diagnostic recall line and is a candidate for removal. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
//--- tradeableOK / selectionScore, not directionalRecallOK / balancedOosEra - see the
//--- SELECTION METRIC note above. The lexicographic shape is unchanged: qualifying
//--- always outranks not qualifying, and the score only breaks ties within a category.
fix: a one-sided era can no longer become the best checkpoint Measured on HYBRID, era 29 of the first win-scored run: the model collapsed to always-Buy and was crowned "new best selection score 67.1%". Under win-based scoring that is not a coincidence - the always-call-the-drift-side model IS the chance reference, so it scores exactly chance (P(winLong) ~ 67% on SP500), while every honest two-sided era scores 63-66% because shorts win less often against the drift. Raw score ranking therefore actively prefers the degenerate model, every regression restores back to it, and live NMS collapses its near-constant signal to ~25 trades per era - observed as "hybrid barely trades". bothSidesLive already blocked one-sided eras from DEPLOYING (tradeableOK, 371f8aa), but among not-yet-deployable eras the score alone ranked - the same early phase the coverage credit was added for, failing the same way through a different door. The ranking key is now three lexicographic tiers: deployable > two-sided > score. A one-sided era cannot displace a two-sided best regardless of score - by construction its score is a property of the data's drift, not the model - and a two-sided era displaces a one-sided best no matter how much lower it scores. m_bestBothSidesLive is snapshotted with the checkpoint and reset with the rest of the best-tracking state. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 21:19:44 -04:00
//---
//--- THREE tiers since 2026-08-09, with bothSidesLive in the middle: (deployable) >
//--- (two-sided) > (score). Forced by a measured failure, not symmetry: HYBRID's era 29
//--- collapsed to always-Buy and scored 67.1% - EXACTLY chance, because under win-based
//--- scoring the degenerate always-call-the-drift-side model IS the chance reference -
//--- while every honest two-sided era scored 63-66% (shorts win less often against
//--- SP500's drift). Raw score ranking crowned it, every regression restored it, and NMS
//--- collapsed its constant signal to ~25 trades/era. One-sidedness already blocked
//--- DEPLOYMENT (tradeableOK), but among not-yet-deployable eras score alone decided.
//--- A one-sided era now cannot displace a two-sided best NO MATTER its score, and a
//--- two-sided era displaces a one-sided best no matter how much lower it scores - by
//--- construction the one-sided score is a property of the DATA's drift, not the model.
feat(ai): rank checkpoints on directional precision, not balanced accuracy Balanced accuracy is maximized by exactly the model this system must never deploy. Measured frontier at fixed signal strength, base rate 6.1%: tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0% tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3% tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5% It rises monotonically as the model calls MORE and is right LESS, because two of its three terms are directional recalls that a call-everything model drives to ~95%, while the Neutral term it sacrifices counts for only a third. The 2026-07-29 run landed exactly there: balanced 58-64% while calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base rate. Only the per-class recall floor stopped those deploying - a guard doing the job the objective should have been doing - and that same guard also rejected the genuinely useful sparse-but-precise checkpoints. Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how many were right. That is what a trading edge is. Two anti-degenerate floors bracket it, since precision alone is trivially maximized by calling almost nothing: coverage must reach a fraction of the true directional base rate (derived, not configured - it adapts to any symbol/timeframe/label rule), and precision must at least beat that base rate. Against the same frontier the deploy order inverts from tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first) to tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage) Balanced accuracy is kept in the log as a diagnostic and marked as such, so a run where the two disagree - the signature of an over-caller - is visible at a glance. MinRecall no longer decides what ships; it now only drives the diagnostic recall line and is a candidate for removal. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
bool isBetterEra = (tradeableOK && !m_bestPassedRecall) ||
fix: a one-sided era can no longer become the best checkpoint Measured on HYBRID, era 29 of the first win-scored run: the model collapsed to always-Buy and was crowned "new best selection score 67.1%". Under win-based scoring that is not a coincidence - the always-call-the-drift-side model IS the chance reference, so it scores exactly chance (P(winLong) ~ 67% on SP500), while every honest two-sided era scores 63-66% because shorts win less often against the drift. Raw score ranking therefore actively prefers the degenerate model, every regression restores back to it, and live NMS collapses its near-constant signal to ~25 trades per era - observed as "hybrid barely trades". bothSidesLive already blocked one-sided eras from DEPLOYING (tradeableOK, 371f8aa), but among not-yet-deployable eras the score alone ranked - the same early phase the coverage credit was added for, failing the same way through a different door. The ranking key is now three lexicographic tiers: deployable > two-sided > score. A one-sided era cannot displace a two-sided best regardless of score - by construction its score is a property of the data's drift, not the model - and a two-sided era displaces a one-sided best no matter how much lower it scores. m_bestBothSidesLive is snapshotted with the checkpoint and reset with the rest of the best-tracking state. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 21:19:44 -04:00
(tradeableOK == m_bestPassedRecall && bothSidesLive && !m_bestBothSidesLive) ||
(tradeableOK == m_bestPassedRecall && bothSidesLive == m_bestBothSidesLive &&
!isFullyCollapsedEra && selectionScore > m_bestBalancedOos);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// The recall-pass-loss clause used to fire on ANY drop out of a full 3-way recall pass,
// even a near-miss on one class at unchanged accuracy (e.g. observed: Buy:56% Sell:41%
// Neutral:34% - Neutral alone missing the 40% floor by a few points) - treating that
// identically to a total collapse back to Neutral-only. With three classes all needing
// to simultaneously clear the floor, that made isWorseEra fire on most eras once a pass
// was ever achieved, ratcheting eta toward ETA_MIN within a handful of eras and then
// (before the recovery bump below existed) leaving it stuck there permanently - visible
// in practice as ~25 back-to-back identical "regressed from best 70.6% to 70.6%" eras.
// Now only counts as worse if accuracy ALSO dropped meaningfully (same threshold
// regardless of whether the recall-pass flag changed too) - losing the recall-pass flag
// at flat/improved accuracy is borderline variance, not a regression worth
// restoring+decaying over.
feat(ai): rank checkpoints on directional precision, not balanced accuracy Balanced accuracy is maximized by exactly the model this system must never deploy. Measured frontier at fixed signal strength, base rate 6.1%: tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0% tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3% tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5% It rises monotonically as the model calls MORE and is right LESS, because two of its three terms are directional recalls that a call-everything model drives to ~95%, while the Neutral term it sacrifices counts for only a third. The 2026-07-29 run landed exactly there: balanced 58-64% while calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base rate. Only the per-class recall floor stopped those deploying - a guard doing the job the objective should have been doing - and that same guard also rejected the genuinely useful sparse-but-precise checkpoints. Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how many were right. That is what a trading edge is. Two anti-degenerate floors bracket it, since precision alone is trivially maximized by calling almost nothing: coverage must reach a fraction of the true directional base rate (derived, not configured - it adapts to any symbol/timeframe/label rule), and precision must at least beat that base rate. Against the same frontier the deploy order inverts from tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first) to tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage) Balanced accuracy is kept in the log as a diagnostic and marked as such, so a run where the two disagree - the signature of an over-caller - is visible at a glance. MinRecall no longer decides what ships; it now only drives the diagnostic recall line and is a candidate for removal. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
bool isWorseEra = selectionScore < m_bestBalancedOos - ETA_DECAY_REGRESSION_PCT;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//--- ENSEMBLE: ranking and checkpointing belong to the ensemble as a unit (see
//--- EnsembleCommitJointCheckpoint). A member's own best era is NOT the deployable one -
//--- committing it here would overwrite the joint checkpoint with a quartet no combined
//--- measurement ever covered, which is the exact failure the ensemble gate exists to
//--- prevent. The eta recovery bump still applies: that is this net's own learning-rate
//--- dynamics, not a deployment decision.
if(isBetterEra && m_ensembleMember)
eta = MathMin(m_etaCeiling, eta / ETA_DECAY_FACTOR);
if(isBetterEra && !m_ensembleMember)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
//--- Snapshot BOTH scores at the checkpoint: m_bestBalancedOos is what ranking compares
//--- against next era; m_bestOosForecast keeps the blended value FinalizeTrainRun() and
//--- the restore branch reset dOosForecast to (see m_bestBalancedOos' declaration).
m_bestOosForecast = dOosForecast;
feat(ai): rank checkpoints on directional precision, not balanced accuracy Balanced accuracy is maximized by exactly the model this system must never deploy. Measured frontier at fixed signal strength, base rate 6.1%: tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0% tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3% tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5% It rises monotonically as the model calls MORE and is right LESS, because two of its three terms are directional recalls that a call-everything model drives to ~95%, while the Neutral term it sacrifices counts for only a third. The 2026-07-29 run landed exactly there: balanced 58-64% while calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base rate. Only the per-class recall floor stopped those deploying - a guard doing the job the objective should have been doing - and that same guard also rejected the genuinely useful sparse-but-precise checkpoints. Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how many were right. That is what a trading edge is. Two anti-degenerate floors bracket it, since precision alone is trivially maximized by calling almost nothing: coverage must reach a fraction of the true directional base rate (derived, not configured - it adapts to any symbol/timeframe/label rule), and precision must at least beat that base rate. Against the same frontier the deploy order inverts from tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first) to tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage) Balanced accuracy is kept in the log as a diagnostic and marked as such, so a run where the two disagree - the signature of an over-caller - is visible at a glance. MinRecall no longer decides what ships; it now only drives the diagnostic recall line and is a candidate for removal. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
m_bestBalancedOos = selectionScore;
m_bestPassedRecall = tradeableOK;
fix: a one-sided era can no longer become the best checkpoint Measured on HYBRID, era 29 of the first win-scored run: the model collapsed to always-Buy and was crowned "new best selection score 67.1%". Under win-based scoring that is not a coincidence - the always-call-the-drift-side model IS the chance reference, so it scores exactly chance (P(winLong) ~ 67% on SP500), while every honest two-sided era scores 63-66% because shorts win less often against the drift. Raw score ranking therefore actively prefers the degenerate model, every regression restores back to it, and live NMS collapses its near-constant signal to ~25 trades per era - observed as "hybrid barely trades". bothSidesLive already blocked one-sided eras from DEPLOYING (tradeableOK, 371f8aa), but among not-yet-deployable eras the score alone ranked - the same early phase the coverage credit was added for, failing the same way through a different door. The ranking key is now three lexicographic tiers: deployable > two-sided > score. A one-sided era cannot displace a two-sided best regardless of score - by construction its score is a property of the data's drift, not the model - and a two-sided era displaces a one-sided best no matter how much lower it scores. m_bestBothSidesLive is snapshotted with the checkpoint and reset with the rest of the best-tracking state. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 21:19:44 -04:00
m_bestBothSidesLive = bothSidesLive;
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
//--- Raw significance inputs for the family-wise gate, taken at the same instant as the
//--- weight snapshot below so the test always describes the weights that would ship.
//--- selectionScore cannot substitute: it is precision x coverage credit, and the test
//--- needs the unweighted precision plus the n that sets its standard error.
m_bestDirPrecPct = dirPrecPct;
m_bestChancePrecPct = chancePrecPct;
m_bestDirCalls = oosDirCalls;
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//--- The operating point is part of the model, not of the run: these OOS numbers were
//--- produced by these weights UNDER this threshold, and restoring one without the
//--- other would deploy a model whose coverage and precision are not the ones the gate
//--- cleared. Captured at the same instant as the weight snapshot below.
m_bestDirConfThreshold = m_dirConfThreshold;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- eval candidates are throwaway - track the score (above) but never write a checkpoint
//--- file; m_haveOosCheckpoint=false then also skips the worse-era RestoreWeights() restore.
//--- In-MEMORY weight snapshot (not a file): the file-based checkpoint re-created every
//--- neuron on restore, which the CPU-DLL backend can't do for a second live set - see
//--- CNet::CaptureWeights/RestoreWeights. eval candidates snapshot nothing.
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
m_haveOosCheckpoint = Net.CaptureWeights();
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// Recovery bump: ETA_DECAY_FACTOR-only ever shrinks eta, and previously nothing ever
// grew it back - a losing streak early in a run (even a since-corrected one) would
// permanently cap how fast every later era could learn for the rest of the run, all
// the way down to ETA_MIN with no way back. A genuinely better era (new best, not
// just a tie) means the current eta is working, so nudge it back up a bit - capped at
// this model's own configured starting rate (m_etaCeiling - AdamLearningRate for
// ADAM, SgdLearningRate for SGD, see that member's declaration comment) so this
// can't runaway past the rate training was actually tuned to start at.
eta = MathMin(m_etaCeiling, eta / ETA_DECAY_FACTOR);
}
else
if(isWorseEra && m_bestOosForecast > 0)
{
// Decaying eta alone only softens FUTURE steps - it does nothing to undo the
// regression this era already baked into the weights, so a run could (and in
// practice did) spend 15+ eras compounding forward from one bad era's damage,
// each new era fighting the last one's overshoot instead of building on the best
// state found so far. Restore the last checkpointed-good weights before continuing
// (mirrors what FinalizeTrainRun() does at the END of a run, just applied live so
// the oscillation can't compound within a single run) - this is what actually turns
// "reduce LR on regression" into "step back, then retry slower", not just "drift
// slower".
//
// BOTH the restore AND the eta decay below are gated on m_bestPassedRecall: before
// ANY era has ever cleared the per-class recall floor, isBetterEra's own
// lexicographic ordering degrades to a pure blended-accuracy tiebreak
// (directionalRecallOK==false on both sides of the comparison), so "best checkpoint"
// during that phase just means "called Neutral most confidently so far" - restoring
// it would actively defend the majority-class collapse against any era that trades
// some accuracy for real Buy/Sell recall, which is exactly the bias this whole
// recall-gate mechanism exists to prevent (see isBetterEra's own comment above).
// Observed in practice: era 1-3 all "improved" on accuracy alone
// (24.9%->41.4%->52.3%) while Buy/Sell recall stayed at a flat 0% the entire time -
// restoring pre-pass would have locked training into that trajectory instead of
// letting it explore past it. Decaying eta has the same bias one step removed:
// every regression relative to a Neutral-collapse "best" shrinks eta a little more,
// steadily strangling the exploration needed to escape that collapse until eta
// bottoms out at ETA_MIN with no real solution ever found and no checkpoint to fall
// back on either - observed in practice as a run whose best-ever blended accuracy
// kept landing on 0%/0%/100% Buy/Sell/Neutral recall eras, each one triggering
// another decay on the very next era, until eta floored out around era 20 and the
// remaining eras just oscillated between collapse states with no way to make a
// large-enough move to escape and no way to reset. Once m_bestPassedRecall is true,
// there IS a genuinely good state worth protecting, and both restoring the
// checkpoint and decaying eta on regression are safe/correct again.
fix(training): escape the recall-gate catch-22 that let runs decay unchecked Evidence (MQL5\Logs, SP500 H1, 2026-07-29): Perceptron era 61 Buy 32% Sell 27% Neut 94% bal 51% LSTM era 160 Buy 16% Sell 11% Neut 98% bal 42% (peaked 49% @ era 44) Hybrid era 179 Buy 5% Sell 2% Neut 99% bal 35% (peaked 41%) CONV era 228 Buy 2% Sell 4% Neut 99% bal 35% (peaked 40% @ era 122) Every model peaks early then decays monotonically toward Neutral, and nothing stops it: the restore-best-weights + decay-eta handler is gated on m_bestPassedRecall, which stays false forever when no checkpoint ever clears the per-class floor. CONV ran 228 eras with eta pinned at its 0.000300 start. The plateau ladder cannot end such a run either (stage 3 refuses to deploy without a recall pass, so it resets ~27 times), making it a 1000-era one-way trip. The gate's own justification had expired. It was written when the pre-pass tiebreak was blended-accuracy-only, where "best" really did mean "called Neutral most confidently". The balanced-selection change replaced that with `balancedOosEra > m_bestBalancedOos` plus an isFullyCollapsedEra exclusion, so a Neutral-only era now scores ~33% - the FLOOR of the balanced metric - and cannot anchor the checkpoint at all. Pre-pass "best" now means "most class-balanced so far", which is worth defending; and isWorseEra is itself a balanced-accuracy regression, so it cannot fire merely for trading Neutral calls for Buy/Sell. The original concern still holds while the best-so-far IS near-collapse, so the escape is margin-guarded: defend the checkpoint only once balanced accuracy sits more than BALANCED_WORTH_DEFENDING_MARGIN_PCT (5pp) above the one-class floor of 100/3. Against the run above that engages for all three stuck topologies (42.3/41.3/50.0 vs a 38.3 threshold) while a genuinely collapsed run still explores freely. Two inputs restored to the regime that actually produced a deploy: - MinRecall 60 -> 40. The one successful auto-deploy in the logs (Hybrid, 28th 00:50, best balanced 66.0%) ran against a 40% floor. 60 has never been shown reachable here - a floor above what the config can reach is the same "target set too high" failure the surrounding comment already warns about. - OversampleParity 60 -> 90. 60 overcorrected. Runs now START Neutral-dominant (Buy 0-11% recall at era 1) and call Buy/Sell on 0-4% of bars against a ~6% true base rate - under-calling, with no headroom to converge down from. The deploying run began at Buy 90% / Sell 36%, 24% of bars called, and settled into the floor from above. Raw over-calling is the intended starting condition; live calls are base-rate-calibrated by AILogitPriorStrength, which is why the input's own note says to judge over-calling by live-fired precision, not raw counts. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 07:51:08 -04:00
// 2026-07-29: the m_bestPassedRecall gate above has an escape now, because its
// stated premise expired. It was written when the pre-pass tiebreak really was
// blended-accuracy-only; the balanced-selection change (m_bestBalancedOos) replaced
// that with `balancedOosEra > m_bestBalancedOos` AND an isFullyCollapsedEra
// exclusion, so a Neutral-only era now scores ~33% (the FLOOR of the balanced
// metric) and cannot anchor the checkpoint at all. "Best checkpoint" pre-pass
// therefore no longer means "called Neutral most confidently" - it means "most
// class-balanced state found so far", which is worth defending, and isWorseEra is
// itself a balanced-accuracy regression, so it cannot fire merely for trading
// Neutral calls for Buy/Sell.
//
// Leaving the gate absolute had a failure mode of its own, and it is not
// hypothetical: if NO checkpoint ever clears the recall floor, m_bestPassedRecall
// stays false forever, so there is never any restore and never any eta decay.
// Observed on SP500 H1 2026-07-29 across three topologies - CONV ran 228 eras with
// eta pinned at its 0.000300 start while balanced accuracy slid 40% -> 35% and Buy
// recall 11% -> 2%. The run had no regression control whatsoever, and the plateau
// ladder could not end it either (stage 3 refuses to deploy without a recall pass),
// so it was a 1000-era one-way trip into a Neutral collapse.
//
// The original concern still applies while the best-so-far IS near-collapse:
// decaying eta against such a "best" strangles the exploration needed to escape it.
// So the escape is margin-guarded - defend the checkpoint only once it sits clearly
// above the one-class floor, which is exactly when there is something real to lose.
bool bestWorthDefending = (m_bestBalancedOos >
BALANCED_COLLAPSE_PCT + BALANCED_WORTH_DEFENDING_MARGIN_PCT);
fix: the recall gate was unsatisfiable and the LR decay was a spiral Both made the run structurally unable to succeed, independently of any signal in the data. Found by reading the 13:01 log. RECALL GATE. m_objectiveMet required Buy, Sell AND Neutral OOS recall each >= 40%. First-touch resolution (ce52654) collapsed Neutral from the ~94% majority it was under exact-pivot labels to a same-bar-tie residue - 250 of 38,261 bars, 0.65% - so the floor was asking the model to identify 40% of coin-flip ties before it could converge. Measured: CONV, LSTM and HYBRID all logged "Neutral:0% (need >=40% each)" on every era. No model could ever satisfy it; every run was destined for the plateau ladder or the era cap. Only the DIRECTIONAL floors are load-bearing for the anti-collapse job the gate exists to do: an all-Neutral model shows Buy and Sell recall at 0% and is blocked by them. Neutral's own floor guarded the mirror bias (over-calling Buy/Sell at Neutral's expense), which was real at 94% prevalence and is not at 0.65% - there, almost never calling Neutral is correct rather than biased. Prevalence-guarded rather than hardcoded off, so it returns by itself if a future label rule makes Neutral substantial again. Deliberately NOT extended to Buy/Sell: exempting a thin directional class reopens the era-44-46 hole, which directionalRecallMeasured only half-covers - it checks those classes were MEASURED, not that they passed. ETA DECAY. A regressing era restored the checkpoint, reset the optimizer and cut eta - all on the FIRST regression. The next era then started from an identical state with a smaller step, regressed again, and got the same treatment. The loop is self-sustaining and cannot discover anything, because rolling the weights back is exactly what removes the exploration that would end it. Measured on PAI: eras 2-11 every one a regression against era 1, eta 0.000594 -> 0.000024, dW/W 0.000%/0.000% from era 2 onward. Ten eras, ~45s each, reproducing era 1 exactly and unable to do anything else. Now requires ETA_DECAY_PATIENCE_ERAS consecutive regressions - the standard ReduceLROnPlateau formulation. A single bad era is noise, and an improving era clears the counter so alternating runs never accumulate into a decay. Build tag -> gate-patience-v3. It had not moved in six commits, which is why the running binary could not be identified from its own log. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:28:58 -04:00
//--- PATIENCE (see ETA_DECAY_PATIENCE_ERAS). Restoring the checkpoint AND cutting
//--- eta on the FIRST regressing era makes the next era start from an identical
//--- state with a smaller step - so it regresses again, and the response to that is
//--- another restore and another cut. The loop is self-sustaining and cannot
//--- discover anything, because rolling the weights back is precisely what removes
//--- the exploration that would end it. Wait for several consecutive regressions
//--- before concluding the step is too big; a single bad era is noise.
m_consecutiveRegressions++;
if((m_bestPassedRecall || bestWorthDefending) &&
m_consecutiveRegressions >= ETA_DECAY_PATIENCE_ERAS)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
fix: the recall gate was unsatisfiable and the LR decay was a spiral Both made the run structurally unable to succeed, independently of any signal in the data. Found by reading the 13:01 log. RECALL GATE. m_objectiveMet required Buy, Sell AND Neutral OOS recall each >= 40%. First-touch resolution (ce52654) collapsed Neutral from the ~94% majority it was under exact-pivot labels to a same-bar-tie residue - 250 of 38,261 bars, 0.65% - so the floor was asking the model to identify 40% of coin-flip ties before it could converge. Measured: CONV, LSTM and HYBRID all logged "Neutral:0% (need >=40% each)" on every era. No model could ever satisfy it; every run was destined for the plateau ladder or the era cap. Only the DIRECTIONAL floors are load-bearing for the anti-collapse job the gate exists to do: an all-Neutral model shows Buy and Sell recall at 0% and is blocked by them. Neutral's own floor guarded the mirror bias (over-calling Buy/Sell at Neutral's expense), which was real at 94% prevalence and is not at 0.65% - there, almost never calling Neutral is correct rather than biased. Prevalence-guarded rather than hardcoded off, so it returns by itself if a future label rule makes Neutral substantial again. Deliberately NOT extended to Buy/Sell: exempting a thin directional class reopens the era-44-46 hole, which directionalRecallMeasured only half-covers - it checks those classes were MEASURED, not that they passed. ETA DECAY. A regressing era restored the checkpoint, reset the optimizer and cut eta - all on the FIRST regression. The next era then started from an identical state with a smaller step, regressed again, and got the same treatment. The loop is self-sustaining and cannot discover anything, because rolling the weights back is exactly what removes the exploration that would end it. Measured on PAI: eras 2-11 every one a regression against era 1, eta 0.000594 -> 0.000024, dW/W 0.000%/0.000% from era 2 onward. Ten eras, ~45s each, reproducing era 1 exactly and unable to do anything else. Now requires ETA_DECAY_PATIENCE_ERAS consecutive regressions - the standard ReduceLROnPlateau formulation. A single bad era is noise, and an improving era clears the counter so alternating runs never accumulate into a decay. Build tag -> gate-patience-v3. It had not moved in six commits, which is why the running binary could not be identified from its own log. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:28:58 -04:00
m_consecutiveRegressions = 0;
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
if(m_haveOosCheckpoint && Net.RestoreWeights())
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
{
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
dOosForecast = m_bestOosForecast;
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//--- The operating point goes back with the weights it was fitted for.
//--- Leaving the current one in place would pair restored weights with a
//--- threshold chosen for the rejected ones - see m_bestDirConfThreshold.
m_dirConfThreshold = m_bestDirConfThreshold;
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//--- 2026-08-09 audit, F3: the snapshot restores WEIGHTS only, so without
//--- this the Adam moments still encode the just-rejected trajectory and
//--- the first updates after the restore push straight back toward the
//--- state that was rolled back - the restore -> regress-again -> restore
//--- oscillation. A restore is a new starting point; it gets a fresh
//--- optimizer.
Net.ResetOptimizerState();
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
if(eta > ETA_MIN)
eta = MathMax(ETA_MIN, eta * ETA_DECAY_FACTOR);
feat(ai): measure precision per confidence tier; fix stale metric labels Two things the 2026-07-30 run exposed. 1. Every user-facing message still called the selection metric "balanced accuracy". It has ranked on directional precision since a142749, so "CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5% PRECISION as if it were macro-recall, while the same era logged an actual balanced accuracy of 49%. Two different numbers under one name, in the line that announces a deploy. Relabelled at every site, including the stage-3 refusal, which still described the per-class recall floor that stopped being the gate. 2. Precision is now bucketed by confidence tier and logged per era, both per-tier and cumulatively from each tier upward: | tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ... The per-tier number says whether confidence is calibrated to correctness at all; if it does not rise T0->T3, raising the floor buys nothing and that is the finding. The ">=" number is what a floor would actually deliver, with its fire count, so the coverage cost is visible in the same line. Tier weights are 25/50/75/100, so for an AI-only config Min_Vote_Open maps straight across: 50 = ">=T1", 75 = ">=T2", 100 = ">=T3". Bucketing happens at the existing live-fired accounting site, so it measures exactly the population that trades - not the raw argmax. Both builds compile 0 errors, 0 warnings. No retrain needed for either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
Print(ID + ": OOS selection score (coverage-weighted dir-precision) regressed from best " + DoubleToString(m_bestBalancedOos, 1) +
fix(ai): report the metric actually compared; surface the derived front-end The plateau/regression line printed balancedOosEra as the current value while comparing against m_bestBalancedOos, which has held the SELECTION score since a142749. Two different metrics in one sentence, so HYBRID logged "regressed from best 14.4% to 34.0%" a hundred times - a regression to a higher number, which is not a thing. The comparison itself was right (selectionScore, coverage weighted, genuinely below best); only the print was wrong. 1039ad9 relabelled these strings but missed that this site passes the wrong variable. The startup config line had the same shape of gap: it printed the dense taper and called itself self-verifying while the DERIVED conv and recurrent stages - the ones that dominate CONV/LSTM/HYBRID - were invisible. It now shows the width into and out of each front-end stage, and flags the case where the dense stack is wider than the vector reaching it (a linear fan-out cannot recover what the bottleneck discarded; it only adds parameters). Flagged, not silently reshaped - that would re-key trained topologies mid-comparison. UsesConvStage()/UsesLstmStage() replace HasConvBeforeLstm() as the primitive, so each subclass declares its composition once and both the capacity budget and the config line derive from it rather than restating it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 15:20:30 -04:00
"% to " + DoubleToString(selectionScore, 1) + "% (blended " + DoubleToString(m_bestOosForecast, 1) +
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
"%->" + DoubleToString(dOosForecast, 1) + "%) - restoring best checkpoint and decaying learning rate to " + DoubleToString(eta, 6));
}
else
feat(logs): throttle the settled per-era diagnostics - measured 22MB/9.5h of confirmed-working systems Measured from the journal (2026-08-19): the era deep-dive line (~2KB) plus the excursion verdict, tier re-rank, calibration move, barrier hold and selection-regressed note each printed EVERY era for EVERY member - ~940 eras/member/day - long after the systems they watch were confirmed working. Yesterday's file was 1.3GB (70% of it the news-filter calendar spam the sweep fix already removed). VerboseMode returns as an INPUT (demoted 2026-08-01 for the marketplace; that track is dead since the 2026-08-16 pivot) and gains a second job: false throttles each settled per-era print to eras 0-3 plus every TRAIN_LOG_EVERY_ERAS-th (25 ~= one deep-dive per ~15min per member); true restores the per-era firehose, flippable live. Never throttled: anything that marks a CHANGE - new bests, restores + eta decays, plateau stage transitions, deploy approvals, warnings, errors, the label-cache/adoption one-shots, and the combined-vote gate line (the active system's primary telemetry, still every era). Semantic fixes over blanket gating: - barrier hold now ARMS silently and prints only when the hold outlasts the 2-min report interval - a brief hold every era is the design, the long hold is the watchdog case the line exists for; - the ensemble deploy REFUSAL prints immediately when its reason changes (that is a finding), on cadence when unchanged; - the filtered-view census prints when its RESULT moves (drawn count, or strongest vote by >=2pp) and at least every 10th sweep. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 09:36:15 -04:00
//--- THROTTLED (2026-08-19): this no-action branch repeated ~600x/day while
//--- noise wandered below a best it was never going to displace. The acting
//--- branch above (restore + eta decay) still always prints - it changes state.
if(TrainLogDue())
Print(ID + ": OOS selection score (coverage-weighted dir-precision) regressed from best " + DoubleToString(m_bestBalancedOos, 1) +
fix(ai): report the metric actually compared; surface the derived front-end The plateau/regression line printed balancedOosEra as the current value while comparing against m_bestBalancedOos, which has held the SELECTION score since a142749. Two different metrics in one sentence, so HYBRID logged "regressed from best 14.4% to 34.0%" a hundred times - a regression to a higher number, which is not a thing. The comparison itself was right (selectionScore, coverage weighted, genuinely below best); only the print was wrong. 1039ad9 relabelled these strings but missed that this site passes the wrong variable. The startup config line had the same shape of gap: it printed the dense taper and called itself self-verifying while the DERIVED conv and recurrent stages - the ones that dominate CONV/LSTM/HYBRID - were invisible. It now shows the width into and out of each front-end stage, and flags the case where the dense stack is wider than the vector reaching it (a linear fan-out cannot recover what the bottleneck discarded; it only adds parameters). Flagged, not silently reshaped - that would re-key trained topologies mid-comparison. UsesConvStage()/UsesLstmStage() replace HasConvBeforeLstm() as the primitive, so each subclass declares its composition once and both the capacity budget and the config line derive from it rather than restating it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 15:20:30 -04:00
"% to " + DoubleToString(selectionScore, 1) + "% (blended " + DoubleToString(m_bestOosForecast, 1) +
fix(training): escape the recall-gate catch-22 that let runs decay unchecked Evidence (MQL5\Logs, SP500 H1, 2026-07-29): Perceptron era 61 Buy 32% Sell 27% Neut 94% bal 51% LSTM era 160 Buy 16% Sell 11% Neut 98% bal 42% (peaked 49% @ era 44) Hybrid era 179 Buy 5% Sell 2% Neut 99% bal 35% (peaked 41%) CONV era 228 Buy 2% Sell 4% Neut 99% bal 35% (peaked 40% @ era 122) Every model peaks early then decays monotonically toward Neutral, and nothing stops it: the restore-best-weights + decay-eta handler is gated on m_bestPassedRecall, which stays false forever when no checkpoint ever clears the per-class floor. CONV ran 228 eras with eta pinned at its 0.000300 start. The plateau ladder cannot end such a run either (stage 3 refuses to deploy without a recall pass, so it resets ~27 times), making it a 1000-era one-way trip. The gate's own justification had expired. It was written when the pre-pass tiebreak was blended-accuracy-only, where "best" really did mean "called Neutral most confidently". The balanced-selection change replaced that with `balancedOosEra > m_bestBalancedOos` plus an isFullyCollapsedEra exclusion, so a Neutral-only era now scores ~33% - the FLOOR of the balanced metric - and cannot anchor the checkpoint at all. Pre-pass "best" now means "most class-balanced so far", which is worth defending; and isWorseEra is itself a balanced-accuracy regression, so it cannot fire merely for trading Neutral calls for Buy/Sell. The original concern still holds while the best-so-far IS near-collapse, so the escape is margin-guarded: defend the checkpoint only once balanced accuracy sits more than BALANCED_WORTH_DEFENDING_MARGIN_PCT (5pp) above the one-class floor of 100/3. Against the run above that engages for all three stuck topologies (42.3/41.3/50.0 vs a 38.3 threshold) while a genuinely collapsed run still explores freely. Two inputs restored to the regime that actually produced a deploy: - MinRecall 60 -> 40. The one successful auto-deploy in the logs (Hybrid, 28th 00:50, best balanced 66.0%) ran against a 40% floor. 60 has never been shown reachable here - a floor above what the config can reach is the same "target set too high" failure the surrounding comment already warns about. - OversampleParity 60 -> 90. 60 overcorrected. Runs now START Neutral-dominant (Buy 0-11% recall at era 1) and call Buy/Sell on 0-4% of bars against a ~6% true base rate - under-calling, with no headroom to converge down from. The deploying run began at Buy 90% / Sell 36%, 24% of bars called, and settled into the floor from above. Raw over-calling is the intended starting condition; live calls are base-rate-calibrated by AILogitPriorStrength, which is why the input's own note says to judge over-calling by live-fired precision, not raw counts. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 07:51:08 -04:00
"%->" + DoubleToString(dOosForecast, 1) + "%) - best so far is still within " +
DoubleToString(BALANCED_WORTH_DEFENDING_MARGIN_PCT, 1) + "pp of the " +
DoubleToString(BALANCED_COLLAPSE_PCT, 1) + "% one-class floor, so there is nothing worth" +
" restoring yet - continuing to explore without decaying eta (still " + DoubleToString(eta, 6) + ")");
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
feat(search): stop on the IN-SAMPLE plateau, and shrink every best-of-K effect before quoting it Points 3 and 4 of the four-point plan. 1. IN-SAMPLE EARLY STOP - and the reason it is worth having is not compute. The plateau ladder stops on the OOS SELECTION score. That is a peek: by the time it fires, every one of those eras has been evaluated out of sample, so all of them sit in the family the deploy gate corrects over (g_ensCandidateEras, Sidak). Training longer therefore does not merely cost time - it RAISES the bar the eventual winner has to clear. The new stop reads the TRAINING error, which the gate never looks at. When the optimiser has stopped improving on data it can see, more eras will not find a better model; they will only enlarge the OOS family. Ending there shrinks the correction, and the shrinkage is legitimate precisely BECAUSE the stopping rule never consulted an out-of-sample number. That distinction is the whole point and it is the one this project has got wrong four times: stop on IS and the family really is smaller; stop on OOS and those eras were searched and still count. Both stops now exist; only this one buys a lower bar. Deliberately more patient than the OOS ladder (IS_ERROR_PATIENCE_MULT = 3x): training error is noisy per era - mini-batch order alone moves it - and ending a run that is still learning costs far more than a few wasted eras. Improvement is RELATIVE (IS_ERROR_IMPROVE_FRAC = 1%), so it does not depend on the loss's absolute scale, and it only acts when a checkpoint exists, since otherwise it would end a run with nothing to deploy. Reset per RUN alongside the ladder, so a resumed run cannot early-stop on its first era against a previous run's best. 2. WINNER'S-CURSE SHRINKAGE ON THE BARRIER-GEOMETRY WINNER. The family-wise permutation gate already establishes that the RANKING is not noise. It says nothing about the SIZE of the winner's effect - and a best-of-K maximum is biased upward by construction, being the largest of K noisy draws. The adoption message quotes that raw maximum and compares it against the incumbent, so the number a reader plans on is the inflated one. The penalty is now measured, not assumed: the same permutation draws that produce the p-value also produce, per draw, the MAXIMUM excess across all candidates under pure noise. The mean of those maxima is exactly what a best-of-K selection is expected to report when there is nothing there. This is the empirical form of the sqrt(2 ln K) x SE penalty the SQX EdgeFinder plugin applies to every maximum it reports (Stats.java:79-88), and it needs no normality assumption because the draws ARE the null distribution. Applied in James-Stein form - effect x max(0, 1 - penalty^2/effect^2) - so a large effect is nearly untouched and a marginal one collapses toward zero. Reported, not gated. The adoption decision still turns on the permutation p-value, which is the right test for "is the ranking real"; the shrunk number is there so the magnitude quoted beside it is one worth planning on. Closes the first of the two EdgeFinder ports identified on 2026-08-12. NOTE on the second EdgeFinder port, deliberately not done here: "let the measurement steer the target" is already true where it matters most - ReportGeometryExpectancyScan ADOPTS the winning barrier geometry under the family-wise gate rather than advising it, and the MI excursion suite publishes a verdict per instrument per config. What is still missing is steering the TRAINING TARGET itself (direction vs excursion) off those verdicts, and that is a design change rather than a surgical one - direction is a closed verdict while excursion SIZE keeps clearing, so the honest version of that change is a target-selection policy, not a flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:29:00 -04:00
//=== IN-SAMPLE ERROR PLATEAU: THE HONEST EARLY STOP ====================================
//--- The ladder below stops on the OOS SELECTION score. That is a peek: the run has by
//--- then evaluated every one of those eras out of sample, so all of them are in the
//--- family the deploy gate must correct over (g_ensCandidateEras, Sidak) - stopping late
//--- does not just cost compute, it RAISES the bar the winner has to clear.
//--- This stop reads the TRAINING error instead, which the gate never looks at. When the
//--- optimiser has stopped making progress on the data it can see, more eras are not
//--- going to find a better model - they only enlarge the OOS family. So ending here
//--- shrinks the correction rather than inflating it, and the shrinkage is legitimate
//--- precisely BECAUSE the stopping rule never consulted an out-of-sample number.
//--- The distinction matters and it is the one this project has got wrong four times:
//--- stop on IS -> the family really is smaller; stop on OOS -> those eras were searched
//--- and still count. Both stops exist; only this one buys a lower bar.
if(dError >= 0.0 && MathIsValidNumber(dError))
{
//--- Relative improvement, so this does not depend on the loss's absolute scale.
if(m_bestIsError < 0.0 || dError < m_bestIsError * (1.0 - IS_ERROR_IMPROVE_FRAC))
{
m_bestIsError = dError;
m_erasSinceBestIsError = 0;
}
else
{
m_erasSinceBestIsError++;
//--- Deliberately more patient than the OOS ladder: training error is noisy per era
//--- (mini-batch order alone moves it), and ending a run that is still learning is
//--- far more expensive than a few wasted eras. Only acts when there is something
//--- to deploy - with no checkpoint this would end the run with nothing to show.
if(m_erasSinceBestIsError >= TrainPlateauPatienceEras() * IS_ERROR_PATIENCE_MULT &&
fix(plateau): the IS-error early stop was inert for every ensemble member 15 hours of training, and the stop that exists to END a run announced itself 1,299 consecutive times without ending anything: SP500 ConvLSTM IN-SAMPLE ERROR PLATEAU - not improved in 1297 / 1298 / 1299 eras (best 0.2689, now 0.3269) ... era 1396, 1397, 1398 SP500 LSTM 536 eras SP500 CONV 442 eras SP500 PAI 150 eras XAUUSD HYB 478 eras XAUUSD LSTM 296 eras XAUUSD CONV 366 eras CAUSE: it wrote its decision into m_plateauStage, and EnsembleEraVerdict mirrors the shared ladder onto every member - `mm.m_plateauStage = g_ensPlateauStage` - on EVERY era, purely so each member's status line shows the collective stage. A display mirror was silently overwriting a decision, so the stop re-armed and re-fired the next era, forever. This is the worst possible direction for this particular bug. Every one of those 1,299 eras was scored out of sample and joined the family the deploy gate corrects over (Sidak, g_ensCandidateEras). The stop's entire purpose is to make that family SMALLER; instead the run spent fifteen hours raising its own bar. - m_isErrorPlateaued: a one-way per-member latch, cleared only by a fresh run. Nothing in the ladder may reset it. The stop condition and the two solo deploy conditions read the latch, not the mirrored stage. - The orchestrator combines: EnsembleEraVerdict requires UNANIMITY across participating members (same participation test the era barrier uses, so an excluded or finished member cannot veto). One member still learning can still move the combined vote, and the vote is what the gate certifies. - Fed in as `dueStage = PLATEAU_STAGE_DEPLOY`, NOT written to g_ensPlateauStage. The block that actually ends the run sits under `dueStage > g_ensPlateauStage`, so assigning the stage directly makes that test false and the deploy never happens - the same inert-write shape as the bug being fixed. Caught before committing; raising dueStage carries it through the ladder's own path (warm restarts skipped, family-wise vote test, measurement screen, joint checkpoint) unchanged. - g_ensIsPlateauAnnounced: announce once per run, not once per era. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 13:09:26 -04:00
m_haveOosCheckpoint && !m_isErrorPlateaued)
feat(search): stop on the IN-SAMPLE plateau, and shrink every best-of-K effect before quoting it Points 3 and 4 of the four-point plan. 1. IN-SAMPLE EARLY STOP - and the reason it is worth having is not compute. The plateau ladder stops on the OOS SELECTION score. That is a peek: by the time it fires, every one of those eras has been evaluated out of sample, so all of them sit in the family the deploy gate corrects over (g_ensCandidateEras, Sidak). Training longer therefore does not merely cost time - it RAISES the bar the eventual winner has to clear. The new stop reads the TRAINING error, which the gate never looks at. When the optimiser has stopped improving on data it can see, more eras will not find a better model; they will only enlarge the OOS family. Ending there shrinks the correction, and the shrinkage is legitimate precisely BECAUSE the stopping rule never consulted an out-of-sample number. That distinction is the whole point and it is the one this project has got wrong four times: stop on IS and the family really is smaller; stop on OOS and those eras were searched and still count. Both stops now exist; only this one buys a lower bar. Deliberately more patient than the OOS ladder (IS_ERROR_PATIENCE_MULT = 3x): training error is noisy per era - mini-batch order alone moves it - and ending a run that is still learning costs far more than a few wasted eras. Improvement is RELATIVE (IS_ERROR_IMPROVE_FRAC = 1%), so it does not depend on the loss's absolute scale, and it only acts when a checkpoint exists, since otherwise it would end a run with nothing to deploy. Reset per RUN alongside the ladder, so a resumed run cannot early-stop on its first era against a previous run's best. 2. WINNER'S-CURSE SHRINKAGE ON THE BARRIER-GEOMETRY WINNER. The family-wise permutation gate already establishes that the RANKING is not noise. It says nothing about the SIZE of the winner's effect - and a best-of-K maximum is biased upward by construction, being the largest of K noisy draws. The adoption message quotes that raw maximum and compares it against the incumbent, so the number a reader plans on is the inflated one. The penalty is now measured, not assumed: the same permutation draws that produce the p-value also produce, per draw, the MAXIMUM excess across all candidates under pure noise. The mean of those maxima is exactly what a best-of-K selection is expected to report when there is nothing there. This is the empirical form of the sqrt(2 ln K) x SE penalty the SQX EdgeFinder plugin applies to every maximum it reports (Stats.java:79-88), and it needs no normality assumption because the draws ARE the null distribution. Applied in James-Stein form - effect x max(0, 1 - penalty^2/effect^2) - so a large effect is nearly untouched and a marginal one collapses toward zero. Reported, not gated. The adoption decision still turns on the permutation p-value, which is the right test for "is the ranking real"; the shrunk number is there so the magnitude quoted beside it is one worth planning on. Closes the first of the two EdgeFinder ports identified on 2026-08-12. NOTE on the second EdgeFinder port, deliberately not done here: "let the measurement steer the target" is already true where it matters most - ReportGeometryExpectancyScan ADOPTS the winning barrier geometry under the family-wise gate rather than advising it, and the MI excursion suite publishes a verdict per instrument per config. What is still missing is steering the TRAINING TARGET itself (direction vs excursion) off those verdicts, and that is a design change rather than a surgical one - direction is a closed verdict while excursion SIZE keeps clearing, so the honest version of that change is a target-selection policy, not a flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:29:00 -04:00
{
Print(ID + ": IN-SAMPLE ERROR PLATEAU - training error has not improved by " +
DoubleToString(100.0 * IS_ERROR_IMPROVE_FRAC, 1) + "% in " +
IntegerToString(m_erasSinceBestIsError) + " eras (best " +
DoubleToString(m_bestIsError, 4) + ", now " + DoubleToString(dError, 4) +
"). The optimiser has stopped learning from the data it CAN see, so further"
" eras cannot find a better model - they would only add candidates to the"
" family the deploy gate corrects over, raising the bar the winner has to"
" clear. Ending the search and deploying the best checkpoint. This stop"
" never read an out-of-sample number, which is what makes the smaller"
" family legitimate rather than a peek.");
fix(plateau): the IS-error early stop was inert for every ensemble member 15 hours of training, and the stop that exists to END a run announced itself 1,299 consecutive times without ending anything: SP500 ConvLSTM IN-SAMPLE ERROR PLATEAU - not improved in 1297 / 1298 / 1299 eras (best 0.2689, now 0.3269) ... era 1396, 1397, 1398 SP500 LSTM 536 eras SP500 CONV 442 eras SP500 PAI 150 eras XAUUSD HYB 478 eras XAUUSD LSTM 296 eras XAUUSD CONV 366 eras CAUSE: it wrote its decision into m_plateauStage, and EnsembleEraVerdict mirrors the shared ladder onto every member - `mm.m_plateauStage = g_ensPlateauStage` - on EVERY era, purely so each member's status line shows the collective stage. A display mirror was silently overwriting a decision, so the stop re-armed and re-fired the next era, forever. This is the worst possible direction for this particular bug. Every one of those 1,299 eras was scored out of sample and joined the family the deploy gate corrects over (Sidak, g_ensCandidateEras). The stop's entire purpose is to make that family SMALLER; instead the run spent fifteen hours raising its own bar. - m_isErrorPlateaued: a one-way per-member latch, cleared only by a fresh run. Nothing in the ladder may reset it. The stop condition and the two solo deploy conditions read the latch, not the mirrored stage. - The orchestrator combines: EnsembleEraVerdict requires UNANIMITY across participating members (same participation test the era barrier uses, so an excluded or finished member cannot veto). One member still learning can still move the combined vote, and the vote is what the gate certifies. - Fed in as `dueStage = PLATEAU_STAGE_DEPLOY`, NOT written to g_ensPlateauStage. The block that actually ends the run sits under `dueStage > g_ensPlateauStage`, so assigning the stage directly makes that test false and the deploy never happens - the same inert-write shape as the bug being fixed. Caught before committing; raising dueStage carries it through the ladder's own path (warm restarts skipped, family-wise vote test, measurement screen, joint checkpoint) unchanged. - g_ensIsPlateauAnnounced: announce once per run, not once per era. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 13:09:26 -04:00
//--- LATCH FIRST, and let the LATCH - not m_plateauStage - be what the deploy
//--- conditions read. m_plateauStage is mirrored from the shared ensemble ladder
//--- on every era (EnsembleEraVerdict), so writing the decision there meant it
//--- survived until the next verdict and no longer. See m_isErrorPlateaued.
m_isErrorPlateaued = true;
feat(search): stop on the IN-SAMPLE plateau, and shrink every best-of-K effect before quoting it Points 3 and 4 of the four-point plan. 1. IN-SAMPLE EARLY STOP - and the reason it is worth having is not compute. The plateau ladder stops on the OOS SELECTION score. That is a peek: by the time it fires, every one of those eras has been evaluated out of sample, so all of them sit in the family the deploy gate corrects over (g_ensCandidateEras, Sidak). Training longer therefore does not merely cost time - it RAISES the bar the eventual winner has to clear. The new stop reads the TRAINING error, which the gate never looks at. When the optimiser has stopped improving on data it can see, more eras will not find a better model; they will only enlarge the OOS family. Ending there shrinks the correction, and the shrinkage is legitimate precisely BECAUSE the stopping rule never consulted an out-of-sample number. That distinction is the whole point and it is the one this project has got wrong four times: stop on IS and the family really is smaller; stop on OOS and those eras were searched and still count. Both stops now exist; only this one buys a lower bar. Deliberately more patient than the OOS ladder (IS_ERROR_PATIENCE_MULT = 3x): training error is noisy per era - mini-batch order alone moves it - and ending a run that is still learning costs far more than a few wasted eras. Improvement is RELATIVE (IS_ERROR_IMPROVE_FRAC = 1%), so it does not depend on the loss's absolute scale, and it only acts when a checkpoint exists, since otherwise it would end a run with nothing to deploy. Reset per RUN alongside the ladder, so a resumed run cannot early-stop on its first era against a previous run's best. 2. WINNER'S-CURSE SHRINKAGE ON THE BARRIER-GEOMETRY WINNER. The family-wise permutation gate already establishes that the RANKING is not noise. It says nothing about the SIZE of the winner's effect - and a best-of-K maximum is biased upward by construction, being the largest of K noisy draws. The adoption message quotes that raw maximum and compares it against the incumbent, so the number a reader plans on is the inflated one. The penalty is now measured, not assumed: the same permutation draws that produce the p-value also produce, per draw, the MAXIMUM excess across all candidates under pure noise. The mean of those maxima is exactly what a best-of-K selection is expected to report when there is nothing there. This is the empirical form of the sqrt(2 ln K) x SE penalty the SQX EdgeFinder plugin applies to every maximum it reports (Stats.java:79-88), and it needs no normality assumption because the draws ARE the null distribution. Applied in James-Stein form - effect x max(0, 1 - penalty^2/effect^2) - so a large effect is nearly untouched and a marginal one collapses toward zero. Reported, not gated. The adoption decision still turns on the permutation p-value, which is the right test for "is the ranking real"; the shrunk number is there so the magnitude quoted beside it is one worth planning on. Closes the first of the two EdgeFinder ports identified on 2026-08-12. NOTE on the second EdgeFinder port, deliberately not done here: "let the measurement steer the target" is already true where it matters most - ReportGeometryExpectancyScan ADOPTS the winning barrier geometry under the family-wise gate rather than advising it, and the MI excursion suite publishes a verdict per instrument per config. What is still missing is steering the TRAINING TARGET itself (direction vs excursion) off those verdicts, and that is a design change rather than a surgical one - direction is a closed verdict while excursion SIZE keeps clearing, so the honest version of that change is a target-selection policy, not a flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:29:00 -04:00
m_plateauStage = PLATEAU_STAGE_DEPLOY;
}
}
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//=== PLATEAU LADDER ====================================================================
//--- Neither branch above fires in the dead zone between "new best" and "regressed by more
//--- than ETA_DECAY_REGRESSION_PCT". This is the response to sitting in it: count eras since
//--- the last new best and escalate. See the PLATEAU_* constants for the full rationale and
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//--- ENSEMBLE: one shared ladder, run by the verdict below, so the members escalate and
//--- finish together instead of drifting into different stages of different searches.
//--- Stash this era's figures first - the last member to arrive needs every member's, and
//--- commits them if the era's VOTE wins. See the ENSEMBLE DEPLOY GATE block.
if(m_ensembleMember)
{
EnsembleStashEraStats(dirPrecPct, chancePrecPct, oosDirCalls, tradeableOK, bothSidesLive,
selectionScore, dOosForecast);
//--- m_eraCount was already incremented at the top of this block, so the era that just
//--- finished - the one the vote buffer is stamped with - is m_eraCount - 1.
EnsembleOosPassComplete(m_eraCount - 1, eta);
}
else
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
if(isBetterEra)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//--- Moving again: retire the ladder AND the restart boost. The recovery bump in the
//--- checkpoint block above has already clamped eta back to at most m_etaCeiling this
//--- era, and that is deliberate now that restarts overshoot the ceiling
//--- (PLATEAU_RESTART_BOOST): the boost exists to kick the run OUT of a basin, and a
//--- new best is the signal it worked - continuing to train at several times the
//--- tuned rate FROM a state worth keeping risks destroying it. The checkpoint just
//--- snapshotted this era regardless. The normal per-era eta schedule takes over.
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
if(m_plateauStage > 0)
Print(ID + ": new best selection score (coverage-weighted dir-precision) " + DoubleToString(m_bestBalancedOos, 1) +
"% - plateau escape worked, clearing plateau stage " + IntegerToString(m_plateauStage));
m_erasSinceBestBalanced = 0;
m_plateauStage = 0;
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
m_restartBoostErasLeft = 0;
fix: the recall gate was unsatisfiable and the LR decay was a spiral Both made the run structurally unable to succeed, independently of any signal in the data. Found by reading the 13:01 log. RECALL GATE. m_objectiveMet required Buy, Sell AND Neutral OOS recall each >= 40%. First-touch resolution (ce52654) collapsed Neutral from the ~94% majority it was under exact-pivot labels to a same-bar-tie residue - 250 of 38,261 bars, 0.65% - so the floor was asking the model to identify 40% of coin-flip ties before it could converge. Measured: CONV, LSTM and HYBRID all logged "Neutral:0% (need >=40% each)" on every era. No model could ever satisfy it; every run was destined for the plateau ladder or the era cap. Only the DIRECTIONAL floors are load-bearing for the anti-collapse job the gate exists to do: an all-Neutral model shows Buy and Sell recall at 0% and is blocked by them. Neutral's own floor guarded the mirror bias (over-calling Buy/Sell at Neutral's expense), which was real at 94% prevalence and is not at 0.65% - there, almost never calling Neutral is correct rather than biased. Prevalence-guarded rather than hardcoded off, so it returns by itself if a future label rule makes Neutral substantial again. Deliberately NOT extended to Buy/Sell: exempting a thin directional class reopens the era-44-46 hole, which directionalRecallMeasured only half-covers - it checks those classes were MEASURED, not that they passed. ETA DECAY. A regressing era restored the checkpoint, reset the optimizer and cut eta - all on the FIRST regression. The next era then started from an identical state with a smaller step, regressed again, and got the same treatment. The loop is self-sustaining and cannot discover anything, because rolling the weights back is exactly what removes the exploration that would end it. Measured on PAI: eras 2-11 every one a regression against era 1, eta 0.000594 -> 0.000024, dW/W 0.000%/0.000% from era 2 onward. Ten eras, ~45s each, reproducing era 1 exactly and unable to do anything else. Now requires ETA_DECAY_PATIENCE_ERAS consecutive regressions - the standard ReduceLROnPlateau formulation. A single bad era is noise, and an improving era clears the counter so alternating runs never accumulate into a decay. Build tag -> gate-patience-v3. It had not moved in six commits, which is why the running binary could not be identified from its own log. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:28:58 -04:00
//--- Patience is about CONSECUTIVE regressions - an era that improves clears it, so a
//--- run that alternates improve/regress never accumulates its way into a decay.
m_consecutiveRegressions = 0;
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
}
else
{
m_erasSinceBestBalanced++;
fix: the Adam second moment was never Adam - all four tiers Root cause of the B=32 regression, and it predates F4 entirely. Every Adam kernel stored v already square-rooted and then fed that stored value back in as if it were the variance: v_new = sqrt(b2 * v_old + (1 - b2) * g^2) That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below unit scale, so the denominator stops tracking the gradient and Adam degrades into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll (batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where a scale-invariant optimizer gives the same distance for both. After the fix all six magnitudes read 1.199 and v tracks |g| exactly. It hit conv/LSTM specifically because they sit behind a batch-norm with running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep in the degraded regime - while the dense stack near the loss stayed in the working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/ 0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already squared v back for gamma/beta and its comment named the kernels as wrong, which is exactly why gamma/beta kept training while the stages behind froze. Persisted .nnw needs no migration - v keeps its std-dev meaning. Also, the two ways F4 exposed it, both mine: - No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods (Krizhevsky 2014; Granziol et al. 2022), applied once in InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD. - Plateau patience denominated in eras, so raising B made the ladder 32x more impatient in its only unit. PAI converged at era 41 on ~49k updates where the same config had been finding new bests at era 1028. TrainPlateauPatienceEras() stretches it by the same sqrt(B). TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23 eras per stage, not 8 -> 45). Both helpers are identities at B=1. Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into the ranking key rather than checked at deploy time so a one-sided era cannot become best-so-far in the first place. Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded CAppDialog teardown that sat ahead of it - the same ordering inversion the rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit (vs ~1.1 s for the three that finished) having reached none of its cleanup, so its arrows stayed on the chart. Steps are now timed in the log. PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now walks every object type and reports the object counts when both are zero. Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt. FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
int dueStage = m_erasSinceBestBalanced / TrainPlateauPatienceEras();
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
if(dueStage > m_plateauStage)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
m_plateauStage = dueStage;
string stageNote = IntegerToString(m_erasSinceBestBalanced) + " eras with no new best selection score (best " +
DoubleToString(m_bestBalancedOos, 1) + "%)";
if(m_plateauStage == PLATEAU_STAGE_RESTART || m_plateauStage == PLATEAU_STAGE_ANNEAL)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//--- BOOSTED WARM RESTART: a plateau needs a bigger step to climb out of its
//--- basin, not a smaller one - and "back to the ceiling" was a NO-OP whenever
//--- the run plateaued without ever tripping the regression decay, because eta
//--- was still AT the ceiling (2026-08-09 audit, F2). Overshoot it instead; the
//--- era-end anneal below walks the rate back to the ceiling over
//--- PLATEAU_PATIENCE_ERAS eras, so this is a bounded kick, not a new permanent
//--- rate. See PLATEAU_RESTART_BOOST for the amplitude rationale.
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
double etaBefore = eta;
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
eta = m_etaCeiling * PLATEAU_RESTART_BOOST;
fix: the Adam second moment was never Adam - all four tiers Root cause of the B=32 regression, and it predates F4 entirely. Every Adam kernel stored v already square-rooted and then fed that stored value back in as if it were the variance: v_new = sqrt(b2 * v_old + (1 - b2) * g^2) That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below unit scale, so the denominator stops tracking the gradient and Adam degrades into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll (batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where a scale-invariant optimizer gives the same distance for both. After the fix all six magnitudes read 1.199 and v tracks |g| exactly. It hit conv/LSTM specifically because they sit behind a batch-norm with running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep in the degraded regime - while the dense stack near the loss stayed in the working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/ 0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already squared v back for gamma/beta and its comment named the kernels as wrong, which is exactly why gamma/beta kept training while the stages behind froze. Persisted .nnw needs no migration - v keeps its std-dev meaning. Also, the two ways F4 exposed it, both mine: - No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods (Krizhevsky 2014; Granziol et al. 2022), applied once in InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD. - Plateau patience denominated in eras, so raising B made the ladder 32x more impatient in its only unit. PAI converged at era 41 on ~49k updates where the same config had been finding new bests at era 1028. TrainPlateauPatienceEras() stretches it by the same sqrt(B). TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23 eras per stage, not 8 -> 45). Both helpers are identities at B=1. Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into the ranking key rather than checked at deploy time so a one-sided era cannot become best-so-far in the first place. Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded CAppDialog teardown that sat ahead of it - the same ordering inversion the rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit (vs ~1.1 s for the three that finished) having reached none of its cleanup, so its arrows stayed on the chart. Steps are now timed in the log. PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now walks every object type and reports the object counts when both are zero. Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt. FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
m_restartBoostErasLeft = TrainPlateauPatienceEras();
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//--- A restart is a new schedule: replaying the plateau's own accumulated Adam
//--- momentum at 5x the rate would retrace the same basin, harder. Fresh moments
//--- make the kick explore instead (audit F3, same mechanism as the
//--- regression-restore reset).
Net.ResetOptimizerState();
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- The focal-gamma anneal that used to accompany this went with focal loss
//--- on 2026-07-31. It was only ever a monotone step toward zero on a second
//--- imbalance correction; the warm restart above is and always was the
//--- actual escape, so both ladder stages keep their distinct patience
//--- thresholds and simply retry the restart.
Print(ID + ": PLATEAU stage " + IntegerToString(m_plateauStage) + " - " + stageNote +
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
". Boosted warm restart: learning rate " + DoubleToString(etaBefore, 6) + "->" + DoubleToString(eta, 6) +
fix: the Adam second moment was never Adam - all four tiers Root cause of the B=32 regression, and it predates F4 entirely. Every Adam kernel stored v already square-rooted and then fed that stored value back in as if it were the variance: v_new = sqrt(b2 * v_old + (1 - b2) * g^2) That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below unit scale, so the denominator stops tracking the gradient and Adam degrades into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll (batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where a scale-invariant optimizer gives the same distance for both. After the fix all six magnitudes read 1.199 and v tracks |g| exactly. It hit conv/LSTM specifically because they sit behind a batch-norm with running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep in the degraded regime - while the dense stack near the loss stayed in the working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/ 0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already squared v back for gamma/beta and its comment named the kernels as wrong, which is exactly why gamma/beta kept training while the stages behind froze. Persisted .nnw needs no migration - v keeps its std-dev meaning. Also, the two ways F4 exposed it, both mine: - No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods (Krizhevsky 2014; Granziol et al. 2022), applied once in InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD. - Plateau patience denominated in eras, so raising B made the ladder 32x more impatient in its only unit. PAI converged at era 41 on ~49k updates where the same config had been finding new bests at era 1028. TrainPlateauPatienceEras() stretches it by the same sqrt(B). TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23 eras per stage, not 8 -> 45). Both helpers are identities at B=1. Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into the ranking key rather than checked at deploy time so a one-sided era cannot become best-so-far in the first place. Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded CAppDialog teardown that sat ahead of it - the same ordering inversion the rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit (vs ~1.1 s for the three that finished) having reached none of its cleanup, so its arrows stayed on the chart. Steps are now timed in the log. PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now walks every object type and reports the object counts when both are zero. Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt. FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
" (annealing back to " + DoubleToString(m_etaCeiling, 6) + " over " + IntegerToString(TrainPlateauPatienceEras()) +
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
" eras), optimizer momentum reset. Best checkpoint is safe - this only changes how the NEXT eras train.");
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
}
else
if(m_plateauStage >= PLATEAU_STAGE_DEPLOY)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- Exhausted: both escapes were tried and neither found a better model, so
//--- this IS the best this configuration reaches. The deploy itself happens in
//--- the era-cap/plateau branch at the TOP of the next era, which reuses the
//--- proven "stop + mark complete -> FinalizeTrainRun restores and deploys the
//--- best checkpoint" path rather than duplicating it here.
//--- Safety: only ever auto-deploys a checkpoint that CLEARED the per-class
//--- recall floor (m_bestPassedRecall). If nothing ever did, there is no model
//--- worth deploying - so the ladder resets and keeps trying instead, leaving
//--- the era cap as the ultimate backstop. That is what stops "train to the
//--- best possible result" from degenerating into "deploy a Neutral collapse".
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
//--- SECOND gate, and the one that matters on a long run: the checkpoint must
//--- survive having been CHOSEN out of every era this run ranked. See
//--- DEPLOY_FAMILY_WISE_ALPHA - the per-era floor alone opens on noise with
//--- probability 1-(1-0.0228)^N, which is 93% by era 112.
double zBest = 0.0, pFam = 1.0;
int nTried = 0;
bool survivesSelection = BestCheckpointSurvivesSelection(zBest, pFam, nTried);
string selectionNote = " | best-of-" + IntegerToString(nTried) + " test: edge " +
DoubleToString(m_bestDirPrecPct - m_bestChancePrecPct, 1) + "pp on " +
IntegerToString(m_bestDirCalls) + " calls = " + DoubleToString(zBest, 2) +
" sigma, family-wise p=" + DoubleToString(pFam, 4) +
" (need <=" + DoubleToString(DEPLOY_FAMILY_WISE_ALPHA, 2) + ")";
fix(consistency): one geometry authority, one exit authority, and the MI screen finally gets a veto Consistency pass before a fresh deployment. Three places where two systems were choosing the same thing and one of them silently lost. 1. THE GEOMETRY SCAN'S DECISION WAS INERT - measured, not suspected. USDJPY, 2026-08-17: 14:24:12.844 adopting barrier geometry 2:8 ... Relabelling and training on it. 14:24:12.979 triple-barrier labels - stop 1.61*ATR, target 3.21*ATR ... this is what trains It adopted 2:8 and trained on 1.61:3.21. ReportBarrierGeometryScan wrote only m_sl_mode/m_tp_mode, and BarrierMultiples ranks the DERIVED pair ABOVE those ints - so on any model carrying a derived pair (every model with a .cfg, including a fresh one whose weights are gone but whose sidecar survived) the adoption changed nothing. Worse, had it changed something it would have been undone immediately: the adoption sets m_labelCachePrebuilt = false, and that prebuild re-runs DeriveBarrierGeometry at era 0, which overwrites m_derivedSl/TpMult from the excursion quantiles. ONE AUTHORITY: the derived pair, because it is what the labels read, what the deploy gate certifies, what g_Derived*AtrMult places on the live order, and what the .cfg pins across restarts. The scan now writes THAT (floored by MIN_SL_ATR_MULTIPLIER, the same floor DeriveBarrierGeometry applies so the live stop can never be wider than the labelled one), republishes to the bridge immediately rather than at the next era end, and forces the sidecar to be rewritten. m_geometryAdopted latches it so the derive pass the adoption itself triggers cannot overwrite it. The scan outranks the derive for an evidential reason, not an architectural one: its winner cleared a permutation test against the null of the MAXIMUM over every eligible pairing, and it scores the incumbent derived pair as a peer in that same field. The derive is a descriptive quantile read with no significance test attached. BEHAVIOURAL CHANGE, and the reason to flag it before a fresh test: barrier geometry will now actually move when the scan says so. Until today it never did. 2. ONE EXIT AUTHORITY, tied to whose certificate the trade was placed under. CheckClosePosition had two routes. The AI early-exit reads the AI vote undiluted and is exactly what the new exit replay reproduces. The blended route thresholds m_direction, the average over EVERY filter including classic ones whose live votes pass 3 never computes - so it can close a position the certificate never modelled, and no replay can ever check it. When g_DerivedSlAtrMult > 0 the AI's measured geometry is on the order, which means the deploy gate's certificate is the reason the trade exists. In that state the AI now governs the exit and the blended route is suppressed. Classic-only configurations are untouched: there the blended route is the only exit opinion and stays exactly as it was. Nothing moves at the shipped defaults either way (Min_Vote_Close = Disabled). 3. EDGEFINDER, SECOND HALF: THE MEASUREMENT NOW STEERS. The MI suite has always printed its verdicts and then trained the direction target regardless of what they said. That gap IS the difference between this and the EdgeFinder discipline: measure what the market offers, THEN aim. m_dirEvidence is set when EITHER the feature/label mutual information OR the normalised excursion asymmetry clears its block-permuted null - an OR, because the two look for the same thing by different routes and requiring both would reject on the weaker of two independent measurements. Normalised asymmetry specifically, never the raw one, which is the volatility confound. Deploy - solo AND ensemble - now requires it. A run without it still trains, and keeps its checkpoint: the research value is real and the measurement can be wrong. It simply may not go live. Reported separately from the statistical gate because the remedy is different: a failed selection test says train differently, this says look somewhere else. Excursion SIZE keeps clearing where direction does not, and that is a risk-control head rather than an entry signal. For the ensemble the check is per-chart by construction - the MI suite runs once and shares its outcome across members - which is the honest treatment: four models finding nothing between them is not four chances at an edge, it is four fits to the same absent information. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:43:20 -04:00
//--- THE SCREEN HAS A VETO. Everything else in this block asks whether the
//--- MODEL is good enough; this asks whether there was anything to find. If
//--- neither the feature/label MI nor the normalised excursion asymmetry
//--- cleared its null, no amount of fitting created directional information -
//--- and every closed direction verdict in this project was reached after
//--- exactly that was attempted anyway. Reported separately from the
//--- statistical gate because the remedy is completely different: a failed
//--- selection test says train differently, this says look somewhere else.
if(!m_dirEvidence)
Print(ID + ": DEPLOY REFUSED BY THE MEASUREMENT SCREEN - " + m_dirEvidenceWhy +
". Neither the feature/label mutual information nor the normalised"
" excursion asymmetry cleared its permutation null on this"
" configuration, so there is no measured directional information here"
" for a model to have learned. The checkpoint is kept and training"
" state is untouched - this is a refusal to go LIVE, not a failure."
" The productive move is a different target or a different market,"
" not more eras: excursion SIZE keeps clearing where direction does"
" not, and that is a risk-control head rather than an entry signal.");
if(m_bestPassedRecall && m_haveOosCheckpoint && survivesSelection && m_dirEvidence)
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
Print(ID + ": PLATEAU stage " + IntegerToString(PLATEAU_STAGE_DEPLOY) + " - " + stageNote +
" across " + IntegerToString(PLATEAU_STAGE_DEPLOY - 1) + " warm restarts. Training has converged on what this"
+ " configuration can reach - deploying the best checkpoint (dir-precision "
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
+ DoubleToString(m_bestBalancedOos, 1) + "%, blended " + DoubleToString(m_bestOosForecast, 1) + "%)."
+ selectionNote + " - CLEARS.");
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
else
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
if(m_bestPassedRecall && m_haveOosCheckpoint)
{
//--- Passed the per-era floor but not the selection correction: this is a
//--- maximum that a pure-noise search of this length produces routinely.
//--- Same verdict the geometry scan and the indicator tuner reach on this
//--- data, and for the same reason - so say so in the same language
//--- instead of implying the model was merely mediocre.
Print(ID + ": PLATEAU stage " + IntegerToString(PLATEAU_STAGE_DEPLOY) + " - " + stageNote +
". The best checkpoint clears the per-era deployability floor but DOES NOT clear the"
+ " null of the MAXIMUM over the eras it was chosen from" + selectionNote +
". A best-of-N this large happens routinely when every era is a noise draw, so the"
+ " ranking carries no evidence of an edge and this model is not safe to trade."
+ " Restarting the plateau ladder and continuing to train; the "
+ IntegerToString(m_maxErasPerRun) + "-era cap remains the backstop.");
m_erasSinceBestBalanced = 0;
m_plateauStage = 0;
}
else
{
Print(ID + ": PLATEAU stage " + IntegerToString(PLATEAU_STAGE_DEPLOY) + " - " + stageNote +
", but no checkpoint has ever cleared the deployability floor (directional calls on" +
fix: the Adam second moment was never Adam - all four tiers Root cause of the B=32 regression, and it predates F4 entirely. Every Adam kernel stored v already square-rooted and then fed that stored value back in as if it were the variance: v_new = sqrt(b2 * v_old + (1 - b2) * g^2) That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below unit scale, so the denominator stops tracking the gradient and Adam degrades into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll (batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where a scale-invariant optimizer gives the same distance for both. After the fix all six magnitudes read 1.199 and v tracks |g| exactly. It hit conv/LSTM specifically because they sit behind a batch-norm with running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep in the degraded regime - while the dense stack near the loss stayed in the working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/ 0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already squared v back for gamma/beta and its comment named the kernels as wrong, which is exactly why gamma/beta kept training while the stages behind froze. Persisted .nnw needs no migration - v keeps its std-dev meaning. Also, the two ways F4 exposed it, both mine: - No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods (Krizhevsky 2014; Granziol et al. 2022), applied once in InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD. - Plateau patience denominated in eras, so raising B made the ladder 32x more impatient in its only unit. PAI converged at era 41 on ~49k updates where the same config had been finding new bests at era 1028. TrainPlateauPatienceEras() stretches it by the same sqrt(B). TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23 eras per stage, not 8 -> 45). Both helpers are identities at B=1. Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into the ranking key rather than checked at deploy time so a one-sided era cannot become best-so-far in the first place. Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded CAppDialog teardown that sat ahead of it - the same ordering inversion the rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit (vs ~1.1 s for the three that finished) having reached none of its cleanup, so its arrows stayed on the chart. Steps are now timed in the log. PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now walks every object type and reports the object counts when both are zero. Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt. FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
" at least a quarter as many bars as actually swing, at a precision above that base rate, with BOTH Buy and Sell"
+ " recall at or above " + DoubleToString(DEPLOY_MIN_SIDE_RECALL_PCT, 0) + "%), so there is nothing safe to"
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
+ " deploy. Restarting the plateau ladder and continuing to train rather than deploying a"
+ " one-class model; the " + IntegerToString(m_maxErasPerRun) + "-era cap remains the backstop.");
m_erasSinceBestBalanced = 0;
m_plateauStage = 0;
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
}
}
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//--- Restart-boost anneal (see PLATEAU_RESTART_BOOST): walk eta geometrically from
//--- boost x ceiling back down to the ceiling over PLATEAU_PATIENCE_ERAS eras, one step
//--- per completed era - the SGDR-style decaying half of the cycle, which is what makes
//--- the boost a bounded kick instead of a new permanent rate. MathMax guards the case
//--- where the regression decay already pulled eta at or below the ceiling mid-window:
//--- the anneal then just expires without fighting it. A new best cleared the counter
//--- above, so a successful escape never reaches here still boosted.
if(m_restartBoostErasLeft > 0)
{
fix: the Adam second moment was never Adam - all four tiers Root cause of the B=32 regression, and it predates F4 entirely. Every Adam kernel stored v already square-rooted and then fed that stored value back in as if it were the variance: v_new = sqrt(b2 * v_old + (1 - b2) * g^2) That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below unit scale, so the denominator stops tracking the gradient and Adam degrades into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll (batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where a scale-invariant optimizer gives the same distance for both. After the fix all six magnitudes read 1.199 and v tracks |g| exactly. It hit conv/LSTM specifically because they sit behind a batch-norm with running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep in the degraded regime - while the dense stack near the loss stayed in the working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/ 0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already squared v back for gamma/beta and its comment named the kernels as wrong, which is exactly why gamma/beta kept training while the stages behind froze. Persisted .nnw needs no migration - v keeps its std-dev meaning. Also, the two ways F4 exposed it, both mine: - No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods (Krizhevsky 2014; Granziol et al. 2022), applied once in InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD. - Plateau patience denominated in eras, so raising B made the ladder 32x more impatient in its only unit. PAI converged at era 41 on ~49k updates where the same config had been finding new bests at era 1028. TrainPlateauPatienceEras() stretches it by the same sqrt(B). TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23 eras per stage, not 8 -> 45). Both helpers are identities at B=1. Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into the ranking key rather than checked at deploy time so a one-sided era cannot become best-so-far in the first place. Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded CAppDialog teardown that sat ahead of it - the same ordering inversion the rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit (vs ~1.1 s for the three that finished) having reached none of its cleanup, so its arrows stayed on the chart. Steps are now timed in the log. PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now walks every object type and reports the object counts when both are zero. Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt. FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
eta = MathMax(m_etaCeiling, eta * MathPow(PLATEAU_RESTART_BOOST, -1.0 / TrainPlateauPatienceEras()));
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
m_restartBoostErasLeft--;
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
m_oosWindow.Add(dOosForecast);
while(m_oosWindow.Total() > STABILITY_WINDOW)
m_oosWindow.Delete(0);
m_oosStable = false;
if(m_oosWindow.Total() >= STABILITY_WINDOW)
{
double oosMin = m_oosWindow.At(0), oosMax = m_oosWindow.At(0);
for(int w = 1; w < m_oosWindow.Total(); w++)
{
oosMin = MathMin(oosMin, m_oosWindow.At(w));
oosMax = MathMax(oosMax, m_oosWindow.At(w));
}
m_oosStable = (oosMax - oosMin) <= STABILITY_TOLERANCE;
}
// The dError<0.1 RMS-error floor is meaningful for the single-neuron regression head
// (m_outputNeuronsCount==1), where it's the only convergence signal available. For the
// 3-neuron one-hot classification head it's redundant with, and far stricter than,
// dOosForecast/directionalRecallOK: reaching RMS error 0.1 across 3 one-hot targets
// needs every output neuron within ~0.17 of its target on average, i.e. near-perfect
// confident calibration on EVERY bar, not just correct argmax calls - unreachable in
// practice under normal market label noise, so classification runs would oscillate
// forever (era after era hitting good OOS accuracy and passing recall, but never
// satisfying this) without this carve-out.
bool errorGateOK = (m_outputNeuronsCount == 3) ? true : (dError < 0.1);
// Convergence (unlike isBetterEra's ranking) FINALIZES the model, so both directional
// classes must have actually been MEASURED this era. An n/a (-1, thin-sample) Buy or
// Sell recall passing directionalRecallOK is deliberate for ranking (early thin
// windows shouldn't deadlock "best" tracking), but letting it pass HERE converges on
// a window that contained no directional bars to disprove the model. Observed
// 2026-07-19: a mid-run label-cache wipe relabeled the whole window Neutral,
// "accuracy" hit 84.9% with Buy/Sell recall both n/a - without this gate a Neutral-only
// model finalizes as a certified success.
bool directionalRecallMeasured = (m_outputNeuronsCount != 3) || (buyRecallPct >= 0 && sellRecallPct >= 0);
//--- VALIDITY of this era's model, no longer "did it hit a target accuracy". The absolute
//--- OOS-accuracy target (the old MinWR input) is gone: an accuracy number typed in ahead of
//--- time is either unreachable for the symbol/timeframe - in which case the run never
//--- converges and burns to the era cap - or set low enough to stop a run that was still
//--- improving. Neither is what "train to the best result" means. Quality is now enforced by
//--- WHICH era gets deployed (balanced-accuracy checkpoint ranking + this per-class recall
//--- floor) and WHEN a run ends (the plateau ladder), not by an accuracy threshold. Note the
//--- recall floor deliberately stays: it is not a performance target but the anti-collapse
//--- gate that makes auto-deploy safe.
m_objectiveMet = errorGateOK && directionalRecallOK && directionalRecallMeasured;
}
// Only mark the persisted model "complete" once it actually converged this era -
// an interruption (stop) or an ordinary in-progress era must stay flagged incomplete
// so a restart resumes training instead of quietly treating a partial run as done.
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
// Convergence = "the plateau ladder is exhausted AND there is a recall-passing checkpoint
// to deploy" - the exact same condition the deploy branch beside the era-cap check uses, so
// the flag written into the .nnw here can never disagree with the decision to stop. While a
// run is still improving (or still has an escape stage left to try) this stays false and the
// per-era save correctly records an in-progress run. Previously this was
// (m_objectiveMet && m_oosStable), which needed the removed absolute accuracy target to mean
// anything: with that target gone m_oosStable alone - just 3 eras inside a 2pp band, which
// is true constantly - would have converged the run at the first flat spot.
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
//--- ...and the family-wise selection gate, for the same reason the deploy branch applies it:
//--- these two conditions MUST stay identical or the flag persisted into the .nnw disagrees
//--- with the decision to stop, and a reload would run inference on a model the ladder had
//--- refused to deploy. Cheap enough to re-evaluate per era (one normal-tail evaluation).
double zConv = 0.0, pConv = 1.0;
int nConv = 0;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//--- ENSEMBLE: the verdict is the ensemble's, so the flag persisted into this member's .nnw
//--- has to be the ensemble's too - otherwise a reload would run one member live against
//--- three still training, which is not the model that was measured. Same
//--- must-stay-identical rule as the solo pair, one level up.
m_trainingComplete = m_ensembleMember
? (g_ensDeployApproved && m_haveOosCheckpoint && m_checkpointEra == g_ensBestEra)
fix(plateau): the IS-error early stop was inert for every ensemble member 15 hours of training, and the stop that exists to END a run announced itself 1,299 consecutive times without ending anything: SP500 ConvLSTM IN-SAMPLE ERROR PLATEAU - not improved in 1297 / 1298 / 1299 eras (best 0.2689, now 0.3269) ... era 1396, 1397, 1398 SP500 LSTM 536 eras SP500 CONV 442 eras SP500 PAI 150 eras XAUUSD HYB 478 eras XAUUSD LSTM 296 eras XAUUSD CONV 366 eras CAUSE: it wrote its decision into m_plateauStage, and EnsembleEraVerdict mirrors the shared ladder onto every member - `mm.m_plateauStage = g_ensPlateauStage` - on EVERY era, purely so each member's status line shows the collective stage. A display mirror was silently overwriting a decision, so the stop re-armed and re-fired the next era, forever. This is the worst possible direction for this particular bug. Every one of those 1,299 eras was scored out of sample and joined the family the deploy gate corrects over (Sidak, g_ensCandidateEras). The stop's entire purpose is to make that family SMALLER; instead the run spent fifteen hours raising its own bar. - m_isErrorPlateaued: a one-way per-member latch, cleared only by a fresh run. Nothing in the ladder may reset it. The stop condition and the two solo deploy conditions read the latch, not the mirrored stage. - The orchestrator combines: EnsembleEraVerdict requires UNANIMITY across participating members (same participation test the era barrier uses, so an excluded or finished member cannot veto). One member still learning can still move the combined vote, and the vote is what the gate certifies. - Fed in as `dueStage = PLATEAU_STAGE_DEPLOY`, NOT written to g_ensPlateauStage. The block that actually ends the run sits under `dueStage > g_ensPlateauStage`, so assigning the stage directly makes that test false and the deploy never happens - the same inert-write shape as the bug being fixed. Caught before committing; raising dueStage carries it through the ladder's own path (warm restarts skipped, family-wise vote test, measurement screen, joint checkpoint) unchanged. - g_ensIsPlateauAnnounced: announce once per run, not once per era. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 13:09:26 -04:00
: ((m_plateauStage >= PLATEAU_STAGE_DEPLOY || m_isErrorPlateaued) && m_bestPassedRecall && m_haveOosCheckpoint
fix(consistency): one geometry authority, one exit authority, and the MI screen finally gets a veto Consistency pass before a fresh deployment. Three places where two systems were choosing the same thing and one of them silently lost. 1. THE GEOMETRY SCAN'S DECISION WAS INERT - measured, not suspected. USDJPY, 2026-08-17: 14:24:12.844 adopting barrier geometry 2:8 ... Relabelling and training on it. 14:24:12.979 triple-barrier labels - stop 1.61*ATR, target 3.21*ATR ... this is what trains It adopted 2:8 and trained on 1.61:3.21. ReportBarrierGeometryScan wrote only m_sl_mode/m_tp_mode, and BarrierMultiples ranks the DERIVED pair ABOVE those ints - so on any model carrying a derived pair (every model with a .cfg, including a fresh one whose weights are gone but whose sidecar survived) the adoption changed nothing. Worse, had it changed something it would have been undone immediately: the adoption sets m_labelCachePrebuilt = false, and that prebuild re-runs DeriveBarrierGeometry at era 0, which overwrites m_derivedSl/TpMult from the excursion quantiles. ONE AUTHORITY: the derived pair, because it is what the labels read, what the deploy gate certifies, what g_Derived*AtrMult places on the live order, and what the .cfg pins across restarts. The scan now writes THAT (floored by MIN_SL_ATR_MULTIPLIER, the same floor DeriveBarrierGeometry applies so the live stop can never be wider than the labelled one), republishes to the bridge immediately rather than at the next era end, and forces the sidecar to be rewritten. m_geometryAdopted latches it so the derive pass the adoption itself triggers cannot overwrite it. The scan outranks the derive for an evidential reason, not an architectural one: its winner cleared a permutation test against the null of the MAXIMUM over every eligible pairing, and it scores the incumbent derived pair as a peer in that same field. The derive is a descriptive quantile read with no significance test attached. BEHAVIOURAL CHANGE, and the reason to flag it before a fresh test: barrier geometry will now actually move when the scan says so. Until today it never did. 2. ONE EXIT AUTHORITY, tied to whose certificate the trade was placed under. CheckClosePosition had two routes. The AI early-exit reads the AI vote undiluted and is exactly what the new exit replay reproduces. The blended route thresholds m_direction, the average over EVERY filter including classic ones whose live votes pass 3 never computes - so it can close a position the certificate never modelled, and no replay can ever check it. When g_DerivedSlAtrMult > 0 the AI's measured geometry is on the order, which means the deploy gate's certificate is the reason the trade exists. In that state the AI now governs the exit and the blended route is suppressed. Classic-only configurations are untouched: there the blended route is the only exit opinion and stays exactly as it was. Nothing moves at the shipped defaults either way (Min_Vote_Close = Disabled). 3. EDGEFINDER, SECOND HALF: THE MEASUREMENT NOW STEERS. The MI suite has always printed its verdicts and then trained the direction target regardless of what they said. That gap IS the difference between this and the EdgeFinder discipline: measure what the market offers, THEN aim. m_dirEvidence is set when EITHER the feature/label mutual information OR the normalised excursion asymmetry clears its block-permuted null - an OR, because the two look for the same thing by different routes and requiring both would reject on the weaker of two independent measurements. Normalised asymmetry specifically, never the raw one, which is the volatility confound. Deploy - solo AND ensemble - now requires it. A run without it still trains, and keeps its checkpoint: the research value is real and the measurement can be wrong. It simply may not go live. Reported separately from the statistical gate because the remedy is different: a failed selection test says train differently, this says look somewhere else. Excursion SIZE keeps clearing where direction does not, and that is a risk-control head rather than an entry signal. For the ensemble the check is per-chart by construction - the MI suite runs once and shares its outcome across members - which is the honest treatment: four models finding nothing between them is not four chances at an edge, it is four fits to the same absent information. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:43:20 -04:00
&& m_dirEvidence
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
&& BestCheckpointSurvivesSelection(zConv, pConv, nConv));
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
double currentIndicatorParams[];
m_indicatorTuner.Flatten(currentIndicatorParams);
if(!Net.Save(m_activeFileName + ".nnw", dError, dUndefine, dForecast, dtStudied, m_activeFileCommon, m_eraCount, m_trainingComplete, currentIndicatorParams))
Print(__FUNCTION__ + ": ERROR - era-end Net.Save failed for " + m_activeFileName + ".nnw (era " + IntegerToString(m_eraCount) + "). Training continues but this era's checkpoint was NOT persisted - a crash/restart now would resume from an older era.");
if(!SaveModelStats(m_activeFileName, m_activeFileCommon)) // keep calibration state paired with the just-saved weights
Print(__FUNCTION__ + ": ERROR - SaveModelStats failed for " + m_activeFileName + " (era " + IntegerToString(m_eraCount) + "). Calibration/online-learning state not persisted this era.");
SaveShadowNet(currentIndicatorParams);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
}
if(shouldLogProgress)
{
string recallInfo = (logBuyRecallPct < 0 && logSellRecallPct < 0 && logNeutralRecallPct < 0) ? "" :
(" | OOS recall Buy:" + (logBuyRecallPct < 0 ? "n/a" : IntegerToString(logBuyRecallPct) + "%") +
" Sell:" + (logSellRecallPct < 0 ? "n/a" : IntegerToString(logSellRecallPct) + "%") +
" Neutral:" + (logNeutralRecallPct < 0 ? "n/a" : IntegerToString(logNeutralRecallPct) + "%") +
feat(gates): derive the recall floor instead of configuring it, publish what the configuration can PROVE, and stop an arbitrary member driving live exits Three changes, all from the same principle: measure what is there before aiming at it, and never certify a number you do not trade. 1. THE RECALL FLOOR IS DERIVED, AND IT MOVES BELOW CHANCE. MinRecall=40 was a constant doing a statistical job. Its reference point is the 33.3% recall a zero-skill 3-class model gets on EVERY class, and against that the constant was accidentally calibrated for exactly one sample size: on USDJPY CONV (n_eff 195) 40% is chance + 2.0 SE; on SP500 PAI (n_eff 42) the same 40% is chance + 0.9 SE. One chart was being held to a bar twice as strict as the other, for no reason anyone chose. CollapseRecallFloorPct() computes it per class from that class's own effective sample - EffectiveSampleSize(), so the overlap deflation the rest of the gates use applies here too - as chance - EDGE_MIN_SIGMAS x SE. 26.5% at n_eff 195, 18.8% at n_eff 42. BELOW chance, deliberately, and this is the substantive change rather than the arithmetic. This gate's only job is refusing to call a COLLAPSED model converged. It is not a quality bar; the deploy gate is the quality bar and it is already rigorous (chance + 2 SE on the deflated sample, Sidak over candidate eras, then the cross-instrument pooled certificate). A convergence gate that ALSO demands provably-above-chance recall on all three classes double-counts that job, and it has failed that way twice here: MinRecall=60 blocked every SP500 H1 run in 2026-07, and the 40 that replaced it made Neutral structurally unreachable once first-touch resolution cut Neutral to a 0.65% residue. A floor nothing can reach does not make a funded account safer, it stops the run converging at all. Testing significantly BELOW chance instead catches what a fixed 40 was actually catching - a model that has stopped emitting a class - and cannot become unreachable by construction. It also fixes the direction the old constant scaled: it now widens on a thin OOS window, where low recall genuinely cannot be told from noise, and tightens on a rich one. Today's SP500 PAI (Buy 51 / Sell 18 / Neutral 30) is still correctly blocked on Sell. This also resolves a standing contradiction the code half-admitted at the isBetterEra comment: selection ranks on coverage-weighted PRECISION while convergence gated on RECALL, so a sparse high-precision abstainer - precisely the model that could clear the deploy bar - was blocked by the floor. The era line now PRINTS the derived floor. Anyone comparing these recalls against a remembered "40" is reading the wrong bar. 2. DETECTABILITY: WHAT THIS CONFIGURATION COULD PROVE, BEFORE IT TRAINS. The DEPLOY BAR line states the bar. It never said what reaching it would take, and that is the actionable direction. ReportDetectability() inverts the same identity - the gate passes when edge >= z x sqrt(p(1-p)/n_eff), so certifying an edge d needs n_eff >= z^2 p(1-p)/d^2 independent calls, hence L times as many raw ones - and prints a +2 / +5 / +10pp ladder as required independent calls, raw calls, and share of the OOS window, marking any rung that needs more than the window holds IMPOSSIBLE. Every term is a property of the CONFIGURATION - geometry via break-even, horizon via mean label lifespan, window via oosCutoff - so no amount of training moves any of them. It fires once, at the first healthy sweep, beside ReportFeatureHealth, for the same reason: that is the first moment the bar grid, the measured geometry and the lifespan are real numbers rather than defaults. It gates nothing. This is the EdgeFinder discipline applied to our own gate: establish what the market and the measurement design have to offer, then point the net at it - rather than spending a thousand eras chasing something this OOS window could never certify. 3. AN ARBITRARY MEMBER WAS DRIVING LIVE EXITS AND TRAILING (user-identified). Every ensemble member ran g_LiveAISignedConfidence = SignedAIConfidence(); unconditionally, every tick. Last writer wins. Its consumers are the AI early-exit route (CExpertSignalCustom::LiveSignedConfidence) and TrailingIntelligent - so on a four-model chart an LSTM entry could be closed, and its stop moved, on the Perceptron's opinion alone, decided by scheduling order. Not the vote, not a weighted blend. Now the mean across registered members, matching how the ensemble actually trades: the open decision is the weighted-average vote, and an abstaining member contributes 0 and dilutes exactly as it does there. Members still training read 0, so a half-trained ensemble reads WEAKER rather than louder - the safe direction for an exit trigger. Deployed and paused members are included, which is the opposite of the era barrier's exemption rule and correct for the opposite reason: that one asks who must be waited for, this asks who has an opinion. Latent today and staying that way for now by choice - Min_Vote_Close ships Disabled (101, unreachable on both scales it drives) and TrailingStrategy is off, so live exits are SL/TP only and the certified hold-to-barrier win rate is what actually gets traded. Fixed now precisely because the plan is to enable vote exits once the models are accurate, at which point a scheduling-order exit would be both harmful and very hard to see. STILL OPEN, and needs a decision before vote exits go on: the member gate and the ensemble vote gate both grade hold-to-barrier, so enabling vote exits makes the certified number stop describing the traded one. Warrior_EA.mq5 currently argues barrier models may keep vote exits because "their label IS the vote's own horizon" - that does not hold, since a vote flip at bar 5 of a 64-bar horizon is not the target-before-stop outcome the gate measured. Either grade the OOS call on the real exit rule (first of SL / TP / vote-flip / horizon) through the fill engine, or set HoldToBarrier for ensemble members so the policy cannot drift from the certificate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:15:08 -04:00
//--- The floor is DERIVED per class now, so it has to be printed rather than assumed - a
//--- reader comparing these recalls against a remembered "40" would be reading the wrong
//--- bar. It is a COLLAPSE floor sitting below the 33.3% chance recall, not a quality bar;
//--- the quality bar is the DEPLOY BAR further along this same line.
StringFormat(" (collapse floor >=%.1f%% each - DERIVED from each class's effective sample,"
" and it sits BELOW the 33.3%% zero-skill recall on purpose: it refuses a"
" COLLAPSED model, it does not certify a good one)", m_lastRecallFloorPct));
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- Balanced accuracy = the checkpoint-selection metric (see m_bestBalancedOos). Shown so the
//--- number the deployed model is actually chosen on is visible next to the recalls it averages.
feat(ai): rank checkpoints on directional precision, not balanced accuracy Balanced accuracy is maximized by exactly the model this system must never deploy. Measured frontier at fixed signal strength, base rate 6.1%: tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0% tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3% tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5% It rises monotonically as the model calls MORE and is right LESS, because two of its three terms are directional recalls that a call-everything model drives to ~95%, while the Neutral term it sacrifices counts for only a third. The 2026-07-29 run landed exactly there: balanced 58-64% while calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base rate. Only the per-class recall floor stopped those deploying - a guard doing the job the objective should have been doing - and that same guard also rejected the genuinely useful sparse-but-precise checkpoints. Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how many were right. That is what a trading edge is. Two anti-degenerate floors bracket it, since precision alone is trivially maximized by calling almost nothing: coverage must reach a fraction of the true directional base rate (derived, not configured - it adapts to any symbol/timeframe/label rule), and precision must at least beat that base rate. Against the same frontier the deploy order inverts from tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first) to tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage) Balanced accuracy is kept in the log as a diagnostic and marked as such, so a run where the two disagree - the signature of an over-caller - is visible at a glance. MinRecall no longer decides what ships; it now only drives the diagnostic recall line and is a candidate for removal. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
//--- Balanced accuracy is retained as a DIAGNOSTIC only - selection ranks on directional
//--- precision now (see the SELECTION METRIC note). Both are shown so a run where they
//--- disagree - the signature of an over-calling model - is visible at a glance.
string balancedInfo = (logBalancedAccPct < 0) ? "" : (" | OOS balanced acc " + IntegerToString(logBalancedAccPct) + "% (diagnostic)");
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
//--- "win-rate", not "dir-precision": since 2026-08-09 this counts calls whose TRADE reached
//--- target before stop, and the chance figure beside it is what always-long/always-short
//--- collected on the same bars. The rename is not cosmetic - the old name described label
//--- agreement, and reading the new number as the old one would understate the model by the
//--- both-won share while overstating its edge against a benchmark that had moved.
fix: the deploy gate graded the un-thresholded model coveragePct, dirPrecPct and the declustered TRADED tally were all computed from oPrevSignal - the RAW argmax - while the live order, the arrow and the panel all run on oDeploySignal, which is argmax AFTER the confidence threshold. The gate was certifying a strategy the EA does not trade. Invisible until now: the threshold sat at ~0.02, so the two populations were the same set. The held-out calibration slice (2189316) moved it to 0.14-0.40 and the gap opened immediately - PAI era 256 graded 100% coverage while its traded population was 21% (3,399 of ~16,200 OOS bars). Consequences that were being hidden: - coveragePct >= minCoveragePct was tested against the wrong population, so a model whose TRADED coverage falls under the 24.8% floor still read as clearing it - precSE = sqrt(p(1-p)/n) used n ~16,000 instead of n ~3,400, so the EDGE_MIN_SIGMAS bar was ~2.2x too lenient on the real evidence - the NMS replay declustered a different, larger stream than live, so threshold-rejected bars consumed cluster slots and set alternation state Gate quantities now read m_oosBuyFired/m_oosSellFired (the thresholded population, already tracked for the live-precision line) and the NMS replay runs on oDeploySignal. The threshold can only turn a direction into Neutral, never flip a side, so the fired set is a strict subset and every per-bar outcome is the one already computed. Recall and logBuyPrecPct deliberately stay on the raw argmax: they measure intrinsic class separation, and thresholding them would conflate "cannot separate the classes" with "declines to act on the separation it found". This is the 9a7c37f defect class, and the NMS block carried a comment warning about it while committing it three lines above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 19:34:32 -04:00
//--- "post-threshold" in the label because the population moved: this counts only the calls that
//--- survive the fitted operating point, which is what the EA trades and what the deploy gate
//--- now ranks. Reading it as the old whole-argmax figure would understate coverage as a
//--- regression when it is the threshold doing its job.
string selectionInfo = (logDirPrecPct < 0) ? " | SELECT: no directional calls survived the threshold" :
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
(" | SELECT win-rate " + IntegerToString(logDirPrecPct) + "% on " +
fix: the deploy gate graded the un-thresholded model coveragePct, dirPrecPct and the declustered TRADED tally were all computed from oPrevSignal - the RAW argmax - while the live order, the arrow and the panel all run on oDeploySignal, which is argmax AFTER the confidence threshold. The gate was certifying a strategy the EA does not trade. Invisible until now: the threshold sat at ~0.02, so the two populations were the same set. The held-out calibration slice (2189316) moved it to 0.14-0.40 and the gap opened immediately - PAI era 256 graded 100% coverage while its traded population was 21% (3,399 of ~16,200 OOS bars). Consequences that were being hidden: - coveragePct >= minCoveragePct was tested against the wrong population, so a model whose TRADED coverage falls under the 24.8% floor still read as clearing it - precSE = sqrt(p(1-p)/n) used n ~16,000 instead of n ~3,400, so the EDGE_MIN_SIGMAS bar was ~2.2x too lenient on the real evidence - the NMS replay declustered a different, larger stream than live, so threshold-rejected bars consumed cluster slots and set alternation state Gate quantities now read m_oosBuyFired/m_oosSellFired (the thresholded population, already tracked for the live-precision line) and the NMS replay runs on oDeploySignal. The threshold can only turn a direction into Neutral, never flip a side, so the fired set is a strict subset and every per-bar outcome is the one already computed. Recall and logBuyPrecPct deliberately stay on the raw argmax: they measure intrinsic class separation, and thresholding them would conflate "cannot separate the classes" with "declines to act on the separation it found". This is the 9a7c37f defect class, and the NMS block carried a comment warning about it while committing it three lines above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 19:34:32 -04:00
IntegerToString(logCoveragePct) + "% of bars (post-threshold)" +
fix: refuse invalid SL/TP, fix the unreachable deploy floor, scale the horizon Three defects found by reading the 2026-08-01 training logs, all of which only became visible because the relabel made the numbers mean something. 1. A STALE ENUM TRAINED FOUR MODELS ON THE WRONG TARGET. `OnInit: trade settings snapshot - SL_Mode=1 TP_Mode=-101` -101 was TP_PREV_SWING, deleted from TAKE_PROFIT_MODE on 2026-07-31 in 7eb48f5. MetaTrader does not validate a saved enum input against the enum's current members, so charts saved before that kept the old integer. BarrierMultiples()'s `if(tpMult <= 0.0) tpMult = slMult;` then quietly turned it into a 1:1 barrier, and all four topologies trained ~250 eras against a strategy nobody selected - while the log reported "target 1.00*ATR" as though it were configured. Since the relabel these two inputs ARE the label definition, so this is not a bad trade setting, it is a wrong dataset. ValidateBarrier- Inputs() now refuses to start (INIT_FAILED + Alert + an explicit fix) on any value that is not an enum member. Members are enumerated rather than range-checked because both enums are sparse and carry negative sentinels, so no min/max test can tell a legal value from a deleted one - which is the entire failure mode. The fallback survives as belt-and-braces but now announces itself: a fallback that cannot say it fired is indistinguishable from correct behaviour. 2. THE DEPLOYABILITY FLOOR BECAME MATHEMATICALLY UNREACHABLE. `tradeableOK` required `dirPrecPct >= baseRatePct`, where baseRatePct is Buy+Sell as a share of all bars. At the old exact-pivot target that was ~6%, so "beat the base rate" read as "beat chance" and the test looked sound. Triple-barrier labels put it at ~83%, so the gate now demanded 83% directional precision - impossible by construction. Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing safe to deploy" at a perfectly healthy 43-45% precision, with no checkpoint able to ship however good it got. Replaced with ZERO-SKILL precision, max(Buy,Sell)/allBars: exactly the score of the degenerate always-call-one-direction model this floor exists to reject. Correct at any base rate - ~43% on the current labels, ~3% on the old rare-pivot ones. The era line now prints "(chance N%, edge +Mpp)" beside the selection score, because 44% precision is excellent against a 3% chance level and worthless against a 43% one, and reading the first as the second is what made tonight's run look better than it was. 3. THE HORIZON IGNORED THE BARRIER GEOMETRY. ComputeBarrierHorizonBars() returned the median ZigZag leg, which measures how long a ~1 ATR move takes and says nothing about how long the CONFIGURED barrier needs. First-passage time out of [-m,+k] scales with m*k, so a 1:3 barrier takes ~3x as long as 1:1; the unscaled horizon would have timed out most 1:3 trades and pushed Neutral straight back up, re-creating the imbalance the relabel removes. Now multiplied by slMult*tpMult, calibrated against a real measurement rather than assumed: the accidental 1:1 run resolved at horizon 12 with only 16.7% timeouts, so the swing median is the right scale at m*k=1. Verifiable, not just asserted: the prebuild now counts barriers that ended on the VERTICAL barrier and reports them as a share of Neutral. Neutral conflates "timed out" with "stopped out" and only the first indicts the horizon. Both builds compile 0 errors / 0 warnings. Forces a retrain - correcting TP_Mode re-keys the fingerprint (|TB:1:-101 -> |TB:1:3), which is right: no existing model was trained on the intended target. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:30:49 -04:00
(logChancePrecPct >= 0
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
? " (chance=break-even " + IntegerToString(logChancePrecPct) + "%, edge " +
fix: refuse invalid SL/TP, fix the unreachable deploy floor, scale the horizon Three defects found by reading the 2026-08-01 training logs, all of which only became visible because the relabel made the numbers mean something. 1. A STALE ENUM TRAINED FOUR MODELS ON THE WRONG TARGET. `OnInit: trade settings snapshot - SL_Mode=1 TP_Mode=-101` -101 was TP_PREV_SWING, deleted from TAKE_PROFIT_MODE on 2026-07-31 in 7eb48f5. MetaTrader does not validate a saved enum input against the enum's current members, so charts saved before that kept the old integer. BarrierMultiples()'s `if(tpMult <= 0.0) tpMult = slMult;` then quietly turned it into a 1:1 barrier, and all four topologies trained ~250 eras against a strategy nobody selected - while the log reported "target 1.00*ATR" as though it were configured. Since the relabel these two inputs ARE the label definition, so this is not a bad trade setting, it is a wrong dataset. ValidateBarrier- Inputs() now refuses to start (INIT_FAILED + Alert + an explicit fix) on any value that is not an enum member. Members are enumerated rather than range-checked because both enums are sparse and carry negative sentinels, so no min/max test can tell a legal value from a deleted one - which is the entire failure mode. The fallback survives as belt-and-braces but now announces itself: a fallback that cannot say it fired is indistinguishable from correct behaviour. 2. THE DEPLOYABILITY FLOOR BECAME MATHEMATICALLY UNREACHABLE. `tradeableOK` required `dirPrecPct >= baseRatePct`, where baseRatePct is Buy+Sell as a share of all bars. At the old exact-pivot target that was ~6%, so "beat the base rate" read as "beat chance" and the test looked sound. Triple-barrier labels put it at ~83%, so the gate now demanded 83% directional precision - impossible by construction. Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing safe to deploy" at a perfectly healthy 43-45% precision, with no checkpoint able to ship however good it got. Replaced with ZERO-SKILL precision, max(Buy,Sell)/allBars: exactly the score of the degenerate always-call-one-direction model this floor exists to reject. Correct at any base rate - ~43% on the current labels, ~3% on the old rare-pivot ones. The era line now prints "(chance N%, edge +Mpp)" beside the selection score, because 44% precision is excellent against a 3% chance level and worthless against a 43% one, and reading the first as the second is what made tonight's run look better than it was. 3. THE HORIZON IGNORED THE BARRIER GEOMETRY. ComputeBarrierHorizonBars() returned the median ZigZag leg, which measures how long a ~1 ATR move takes and says nothing about how long the CONFIGURED barrier needs. First-passage time out of [-m,+k] scales with m*k, so a 1:3 barrier takes ~3x as long as 1:1; the unscaled horizon would have timed out most 1:3 trades and pushed Neutral straight back up, re-creating the imbalance the relabel removes. Now multiplied by slMult*tpMult, calibrated against a real measurement rather than assumed: the accidental 1:1 run resolved at horizon 12 with only 16.7% timeouts, so the swing median is the right scale at m*k=1. Verifiable, not just asserted: the prebuild now counts barriers that ended on the VERTICAL barrier and reports them as a share of Neutral. Neutral conflates "timed out" with "stopped out" and only the first indicts the horizon. Both builds compile 0 errors / 0 warnings. Forces a retrain - correcting TP_Mode re-keys the fingerprint (|TB:1:-101 -> |TB:1:3), which is right: no existing model was trained on the intended target. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:30:49 -04:00
(logDirPrecPct - logChancePrecPct >= 0 ? "+" : "") +
IntegerToString(logDirPrecPct - logChancePrecPct) + "pp)"
: ""));
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//--- The operating point that produced the coverage figure just above it, so the two are read
//--- together: coverage falling is only good news if it is this that caused it.
selectionInfo += " @margin>=" + DoubleToString(m_dirConfThreshold, 2);
fix: NMS gates the TRADE, not just the arrow - one arrow is now one trade NmsLiveAccept() appeared in exactly one place: wrapped around DrawObject(). It never touched dPrevSignal, and dPrevSignal is what LongCondition() / ShortCondition() / SignedAIConfidence() read. So a declustered bar lost its arrow and still opened a position. Measured on SP500 H1 2026-08-09: CONV called a direction on 64% of bars, so the ~500 bars visible on screen held ~320 decisions - and ~40 arrows were drawn. Roughly one arrow per eight positions the EA would take. And the survivors are not a random eighth. Rule 2 of the declustering keeps the HIGHER-CONFIDENCE side of a cluster, so the visible set is systematically the best member of each run. A chart showing the best of every eight decisions and hiding the rest reads far better than the model is - the same best-of-N selection error already corrected in the geometry scan, the indicator tuner, the lag profile and the deploy gate, this time on the display layer, where it is most likely to mislead the person deciding whether to trade. Fixed by neutralising dPrevSignal when NMS rejects, rather than adding a "may trade" flag consulted at each read site: that leaves exactly ONE definition of what the model decided this bar, so the arrow, the panel's "Current signal", the confidence feeding sizing/SL/TP/trailing, the refresh tally and the order itself cannot drift apart again. Also reports the consequence instead of hiding it. Every OOS counter on the era line still scores every directional call - a population ~8x larger than what now trades - so the line carries a second figure: | TRADED (declustered) NN% on N calls (edge +Npp) replaying the identical rule over pass 3 (which walks OOS bars oldest to newest, the same order the live sweep sees). Its cursors are separate members from the live ones so a training pass can never disturb the live chart's declustering. Deliberately NOT switched into selectionScore yet. Declustering cuts coverage from ~64% of bars to ~8%, well under MIN_COVERAGE_FRACTION_OF_BASE_RATE, which would make every checkpoint undeployable overnight - the minRR collision and the recall-floor catch-22 twice over. The floor gets re-derived from these measurements first. Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-09 10:22:31 -04:00
//--- TRADED precision: the same calls after declustering, which since 2026-08-09 is exactly the
//--- set that becomes positions (live NMS gates the trade, not just the arrow). Printed next to
//--- the figure it corrects rather than replacing it, because the two answer different questions
//--- - "how good is the model's directional call" vs "how good are the trades it would take" -
//--- and the gap between them is itself the diagnostic. Compare against the SAME chance rate:
//--- declustering changes which bars are called, not what a no-skill model would score on them.
if(m_signalClusterWindow > 0 && m_oosNmsFired > 0)
{
int nmsPrec = (int)MathRound(100.0 * m_oosNmsHits / m_oosNmsFired);
selectionInfo += " | TRADED (declustered) " + IntegerToString(nmsPrec) + "% on " +
IntegerToString(m_oosNmsFired) + " calls" +
(logChancePrecPct >= 0
? " (edge " + (nmsPrec - logChancePrecPct >= 0 ? "+" : "") +
IntegerToString(nmsPrec - logChancePrecPct) + "pp)"
: "");
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// See logBuyPredPct's declaration comment for why this is worth logging alongside recall -
// it's what tells apart a suppressed/dead output (predicted rate stuck at 0%) from a
// miscalibrated boundary (predicted rate healthy, precision poor), which look identical from
// recall alone.
string predictedInfo = (logBuyPredPct < 0 && logSellPredPct < 0) ? "" :
(" | OOS calls Buy:" + (logBuyPredPct < 0 ? "n/a" : IntegerToString(logBuyPredPct) + "%") +
" (win rate " + (logBuyPrecPct < 0 ? "n/a" : IntegerToString(logBuyPrecPct) + "%") + ")" +
" Sell:" + (logSellPredPct < 0 ? "n/a" : IntegerToString(logSellPredPct) + "%") +
" (win rate " + (logSellPrecPct < 0 ? "n/a" : IntegerToString(logSellPrecPct) + "%") + ")");
//--- Live-fired precision: the number that actually predicts forward-trading performance - only
//--- the directional calls that cleared the confidence floor under the live/prior-corrected rule
//--- (see AdjustedSignalFromSoftmax). Count in parentheses = how many bars the model would have
//--- traded this era. "0" fires = the calibration is (this era) suppressing all directional trades.
string liveInfo = (m_lastBuyFired <= 0 && m_lastSellFired <= 0) ? " | live fires 0 this era" :
(" | live win rate Buy:" + (logBuyFiredPrecPct < 0 ? "n/a" : IntegerToString(logBuyFiredPrecPct) + "%") +
" (" + IntegerToString(m_lastBuyFired) + ")" +
" Sell:" + (logSellFiredPrecPct < 0 ? "n/a" : IntegerToString(logSellFiredPrecPct) + "%") +
" (" + IntegerToString(m_lastSellFired) + ")");
feat(ai): measure precision per confidence tier; fix stale metric labels Two things the 2026-07-30 run exposed. 1. Every user-facing message still called the selection metric "balanced accuracy". It has ranked on directional precision since a142749, so "CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5% PRECISION as if it were macro-recall, while the same era logged an actual balanced accuracy of 49%. Two different numbers under one name, in the line that announces a deploy. Relabelled at every site, including the stage-3 refusal, which still described the per-class recall floor that stopped being the gate. 2. Precision is now bucketed by confidence tier and logged per era, both per-tier and cumulatively from each tier upward: | tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ... The per-tier number says whether confidence is calibrated to correctness at all; if it does not rise T0->T3, raising the floor buys nothing and that is the finding. The ">=" number is what a floor would actually deliver, with its fire count, so the coverage cost is visible in the same line. Tier weights are 25/50/75/100, so for an AI-only config Min_Vote_Open maps straight across: 50 = ">=T1", 75 = ">=T2", 100 = ">=T3". Bucketing happens at the existing live-fired accounting site, so it measures exactly the population that trades - not the raw argmax. Both builds compile 0 errors, 0 warnings. No retrain needed for either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
//--- Precision BY CONFIDENCE TIER, and cumulatively from each tier upward - the two numbers a
//--- decision about Min_Vote_Open actually needs. The per-tier figure says whether confidence is
//--- calibrated to correctness at all (it should rise T0->T3; if it does not, raising the floor
//--- buys nothing and the finding is that the head's confidence is uninformative). The ">=Tn"
//--- figure is what you would ACTUALLY get, because a floor keeps every tier at or above it, and
//--- it comes with the fire count so the coverage cost of raising the floor is visible in the
//--- same line. Tier weights are 25/50/75/100, so for an AI-only config the input maps straight
//--- across: Min_Vote_Open 50 = ">=T1", 75 = ">=T2", 100 = ">=T3".
string tierInfo = "";
int tierFiredTotal = 0;
for(int ti = 0; ti < 4; ti++)
tierFiredTotal += m_oosTierFired[ti];
if(tierFiredTotal > 0)
{
tierInfo = " | tier prec";
for(int ti = 0; ti < 4; ti++)
{
int cumFired = 0, cumHits = 0;
for(int tj = ti; tj < 4; tj++)
{
cumFired += m_oosTierFired[tj];
cumHits += m_oosTierHits[tj];
}
tierInfo += " T" + IntegerToString(ti) + ":" +
(m_oosTierFired[ti] > 0
? IntegerToString((int)MathRound(100.0 * m_oosTierHits[ti] / m_oosTierFired[ti])) + "%"
: "n/a") +
"(" + IntegerToString(m_oosTierFired[ti]) + ")" +
(cumFired > 0
? "[>=" + IntegerToString((int)MathRound(100.0 * cumHits / cumFired)) + "%/" +
IntegerToString(cumFired) + "]"
: "");
}
}
feat(ai): report per-layer weight movement each era Adds "dW/W dense1:0.412(0.31%) conv1:0.088(0.000%) ..." to the era line: each layer's weight L2 norm and its relative change since the previous era. Why: a frozen stage and a badly-suited architecture look identical from the outside. Both give a flat metric and a retreat to the majority class, and neither the loss, the accuracy nor the per-class recall can tell them apart. This session cost two full retrain cycles guessing between them - a forget- gate bias (a real bug, measured, but not the cause of the observed failure) and a conv receptive field (which turned out to be a regression, not a fix). A layer sitting at ~0.000% era after era while its neighbours move is receiving no gradient, and no amount of retraining or hyperparameter work will change that. A net where every layer moves and the output still collapses is a genuine architecture or objective problem. The distinction is one glance at the log instead of a redeploy-and-wait cycle per hypothesis. Costs one host-side buffer read per layer per era, off the training path. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 07:10:09 -04:00
// Per-layer weight movement. Pairs with rawOutInfo below: a collapsed constant-classifier state
// has two very different causes, and only this tells them apart. If every layer moves and the
// output still collapses, the architecture or the objective is at fault; if one stage sits at
// ~0.000% era after era while the others move, that stage is receiving no gradient and no amount
// of retraining or hyperparameter work will help. See CNet::LayerLearningReport.
string layerInfo = (CheckPointer(Net) == POINTER_INVALID) ? "" :
(" | dW/W" + Net.LayerLearningReport());
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// Raw-output saturation diagnostic - see m_oosOutMin's declaration comment. Spread ~0 with
// all six min/max values pinned together = the collapsed constant-classifier state.
string rawOutInfo = (m_oosOutCount <= 0) ? "" :
StringFormat(" | OOS raw out B:%.3f..%.3f S:%.3f..%.3f N:%.3f..%.3f spread avg %.4f",
m_oosOutMin[0], m_oosOutMax[0], m_oosOutMin[1], m_oosOutMax[1],
m_oosOutMin[2], m_oosOutMax[2], m_oosOutSpreadSum / m_oosOutCount);
feat(diagnostics): split a reported "Neutral" into CHOSE vs TIED - they need opposite fixes ApplyClassificationSoftmax() requires a STRICT majority over both rivals and sends every tie, 2-way or 3-way, to Neutral. So "OOS recall Neutral:100%" is two completely different events sharing one label: CHOSE - the net genuinely ranks Neutral highest. A class-prior/label problem. TIED - the top two are EXACTLY equal, so the net expressed no preference and the tie-break reported Neutral. A SATURATION problem: the head is SIGMOID, and a saturated sigmoid returns exactly 0.0f or 1.0f in the DLL's float32, so two classes pinned to the same rail compare equal and the bar is silently discarded. Nothing in the logs could tell them apart, and the fixes point opposite ways. Eras 1-25 of the 2026-08-17 solo PAI run read "Neutral 100%" at spread avg 0.99 - fully saturated - and broke out at era 27 as the spread fell to 0.75. That is consistent with EITHER story. The user reports the Neutral phase on most runs, so it is worth four longs to stop guessing. Four per-era counters on the pass 3 OOS walk, reported as: | Neutral CHOSE 12.4% / TIED 38.1% (of which B=S 1204) | rail 61.2% m_oosNeutralStrict - Neutral strictly highest m_oosNeutralTie - no strict winner; the tie-break produced Neutral m_oosTieBuySell - the costly subset: Buy and Sell tied AT the top, i.e. a DIRECTIONAL reading thrown away by float equality m_oosRailBars - any raw output sitting on a sigmoid asymptote, the saturation that makes exact ties possible at all Read on the RAW logits, before ApplyClassificationSoftmax() overwrites TempData in place. Legitimate because softmax is strictly monotone: it cannot change the ordering and cannot break a tie either, so the raw reading and the decision always agree. Placed alongside the existing min/max/spread capture so all the output diagnostics describe the same values. Measurement only - no decision path reads these. NOT COMPILED - user compiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:06:54 -04:00
//--- Splits a reported "Neutral" into the two events that share that label. CHOSE = the net
//--- ranks Neutral highest (a class-prior problem); TIED = the top two are exactly equal and
//--- the tie-break reported Neutral (a saturation problem). These need OPPOSITE fixes, and
//--- until now nothing in the logs could tell them apart. 'of which B=S' is the costly subset -
//--- a directional reading thrown away by float equality - and 'rail' is the saturation that
//--- makes exact ties possible at all. See m_oosNeutralStrict's declaration comment.
fix(batchnorm): bound the normalized value - a constant input feature was amplified 1e4x and pinned PAI's head to its rails BN_MIN_STD = 1e-4 caps the per-unit gain at 1/1e-4 = 1e4, and the comment above it states that as though it were a safety property. It is not. A unit whose running variance is ~0 is a CONSTANT feature carrying no information, and dividing its rounding noise by 1e-4 hands the next layer an activation of several hundred. BN's contract is "output has ~unit variance"; a unit that cannot supply that must contribute nothing, not the largest signal in the layer. MEASURED, 2026-08-17 SP500 H4, four topologies on identical separate charts: model spread Neutral CHOSE Neutral TIED rail CONV 0.386 0.68% 0.10% 0.48% LSTM 0.392 0.63% 0.00% 0.00% HYB 0.376 1.79% 0.00% 0.01% PAI 0.192 0.09% 80.63% 99.99% bn1's cached nx normed 1.38e4 over 800 units. PAI's SIGMOID head was on its rails on 99.99% of bars, with Buy and Sell landing on the SAME rail so they compared exactly equal, and ApplyClassificationSoftmax()'s strict-majority rule reported that tie as Neutral on ~80% of bars. So the long-running "PAI is heavily biased toward Neutral" was never a class-prior problem: the net CHOSE Neutral on 0.09% of bars. It was float equality on a saturated head. The 331ab29 counters answered it on their first run. PAI-only because it is the one topology whose FIRST batch norm sits on the raw 800-dim input vector - CONV/LSTM/CONVLSTM all have a conv or LSTM stage in front, so their first BN sees a learned representation with no degenerate units. That asymmetry was already on file as a suspicion; this is the mechanism. FIX, mirrored in both backends (host NeuronBatchNorm.mqh and device Network.cl): forward nx = clamp(delta/sd, -BN_MAX_NX, +BN_MAX_NX), BN_MAX_NX = 8 backward if the forward bound this unit, the output stopped depending on the input, so d(nx)/dx = 0 and NO gradient passes The backward half is not optional. g is divided by the same sd the forward multiplies by, so a degenerate unit gets its GRADIENT amplified 1e4x too - the "receives gradients divided by sqrt(var) ~ 500" pathology already noted in Network.cl's Adam kernel. Bounding only the forward would move the explosion downstream. 8 sigma is inert on anything healthy (|nx| > 8 is a ~1e-15 event under normality); it binds only on degenerate units, which is the entire point. Same clamp-to-range idiom the activation derivatives beside it already use. SelfCheckBnForward/SelfCheckBnHiddenGrad already prove host against kernel, and BN_OPT_NX was already consumed in the backward for the gamma gradient, so the new read adds no lifetime assumption. Expect PAI to change behaviour and CONV/LSTM/CONVLSTM not to (their rail rate is ~0%, so the clamp never binds). No .nnw format or fingerprint change. ALSO: print the zero-skill reference on the era line. m_oosWinLongTotal and m_oosWinShortTotal have been accumulated for a long time and NEVER printed, which is why three separate topologies all sitting at 62% read as a mysterious coincidence rather than the obvious base rate. It is not a coincidence: with the target (1.70 ATR) nearer than the stop (3.07 ATR), BOTH sides win on 24.5% of bars, so winLong+winShort covers ~124% of them and a no-edge caller collects 62.1% whichever way it calls - against a 64.3% break-even. Derived from this run's own label counts: (11329 - 2776 + 2*2776) / (2*11350) = 62.14%. Every win rate on that line must be read against this, not against 50%. NOT COMPILED - user compiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:21:51 -04:00
//--- THE ZERO-SKILL REFERENCE, measured on THIS era's own OOS bars. Both accumulators have
//--- existed for a while and neither was ever printed, which is why a 2026-08-17 run of three
//--- separate topologies all sitting at 62% read as a mysterious coincidence instead of as the
//--- obvious base rate. It is not a coincidence: with the measured geometry putting the target
//--- (1.70 ATR) nearer than the stop (3.07 ATR), BOTH sides win on ~24.5% of bars, so
//--- winLong + winShort covers ~124% of them and a caller with no edge collects ~62% whichever
//--- way it calls. Every win rate on this line has to be read against this number, not against
//--- 50% and not against the label frequency - see m_oosWinLongTotal for the 30pp-short models
//--- the deploy gate cleared back when it benchmarked one against the other.
//--- The gate itself ranks on MAX(long, short) (see chancePrecPct); both are shown here because
//--- the SPREAD between them is the directional drift, and a model that merely reproduces it
//--- has found the drift, not an edge.
fix(geometry): the target was small BY CONSTRUCTION - ratio is now policy, scale is measured, ladder ceiling removed The derivation read the stop from q75 of ADVERSE travel and the target from q50 of FAVOURABLE travel. Over one horizon those distributions are broadly the same shape, so q75 > q50 MECHANICALLY - the target came out smaller than the stop no matter what the market did. SP500 H4 shipped stop 3.07 / target 1.70: a 0.55:1 payoff needing 64.3%. That was never a measurement, it was two mismatched constants. The reachability line printed beside it - "target on 50.0% of bars, stop on 25.0%" - is exactly 1-q50 and 1-q75. Tautological. It cannot disconfirm anything, and it read as validation. WIDTH AND RATIO ARE INDEPENDENT AND ONLY ONE PAYS. EV = edge x width; ratio is EV-neutral (a driftless walk reaches +m before -k with probability k/(k+m), which IS break-even). Width is what buys cost efficiency: the spread is a fixed 0.047*ATR here, so the shipped 4.77*ATR width paid it 21 times per unit of travel. So: RATIO = policy. BARRIER_TARGET_RR = 2.0 (user's 1:2). Break-even 33.3%. SCALE = measured. The stop quantile is chosen from a ladder, WIDEST FIRST, taking the first rung whose implied 2x target is still reached often enough to be a trainable class. That last clause is the difference from the min-reward:risk raise removed in 2026-08-09, which forced target = 2 x stop with NO reachability test, landed on 6.66*ATR reachable on 3.3% of bars, and trained the model to predict something that essentially never happened. Same ratio; the scale now retreats until the data says the target is attainable. Every rung is logged. LADDER CEILING REMOVED. BARRIER_LADDER stopped at 5.00 and the expectancy scan's "best resolvable pair on width alone" came back as stop 5.05 / target 4.95 - it pinned to the top rung. A recommendation landing exactly on the edge of its own search space is a boundary, not a finding: it cannot tell "5 ATR is optimal" from "5 ATR is all we allowed". Extended to 20*ATR (8 -> 14 rungs). Nothing else needs editing - every consumer is parameterised by BARRIER_LADDER_COUNT - and the horizon constraints (decided >= 60%, reachability floor) now bind instead of a constant. THE SCAN COULD NOT SEE THE SHIPPED GEOMETRY. ReportBarrierGeometryScan looked the configured pair up in its integer grid, and DeriveBarrierGeometry produces CONTINUOUS multiples (3.07/1.70) that can never equal a grid point - so cfgExcess stayed at its -1.0 sentinel and the report printed "configured 3:2 scores -1.00000", which reads as a catastrophic score and actually means "never evaluated". Worse, the grid skipped target<stop entirely because it "inverts the trade's whole premise" - while the derivation was shipping exactly that. The incumbent is now always scored as a peer (never crowned; it is already in force and is not an enum pairing the scan could adopt). BREAK-EVEN NOW INCLUDES THE SPREAD. Every report quoted the frictionless SL/(SL+TP). On SP500 H4 that read 64.3% while the MEASURED zero-skill rate was 62.1% - a 2.2pp gap that IS the cost, and that made every model look 2.2pp better than it was. CostAdjustedBreakEvenPct() prices a win at (TP - spread) and a loss at (SL + spread), matching the expectancy scan's convention exactly so the two reports cannot disagree. It also feeds FitDirConfThreshold, which is the correctness half: the operating point subtracts break-even from precision, so the frictionless figure made every candidate threshold look better by the width of the spread - 2.2pp against a measured edge of 2.3pp, i.e. very nearly all of it. Era line now carries both: "break-even 64.3% frictionless, 66.6% AFTER SPREAD". Forces a full relabel and retrain. Requested. NOT COMPILED - user compiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:34:32 -04:00
double beFrictionless = 50.0;
{
double slBe = 0.0, tpBe = 0.0;
BarrierMultiples(slBe, tpBe);
if(slBe + tpBe > 0.0)
beFrictionless = 100.0 * slBe / (slBe + tpBe);
}
feat(measurement): fix zero-skill denominator, publish the deploy bar, measure lifespan per rung, add a MEASURE scale objective The last run could not have demonstrated an edge either way, and nothing in the log said so. Four changes so it does. 1. THE ZERO-SKILL LINE DIVIDED BY THE WRONG DENOMINATOR. m_oosWinLongTotal resets every era; m_oosSamples only resets on a full model reset. So 'always-long %' decayed as ~1/era: a run whose true rate is 37% printed 1.2% at era 33 and 0.0% at era 2219. This is the SAME bug already found and fixed for logBuyPredPct thirty lines above ('era-15 Buy:2% that was really ~30%'), left in the one line whose whole job is to be the reference every other number is read against. Correct at era 1, wrong everywhere after - including the '62% zero-skill' figure in the 2026-08-16 notes. Now per-era, and always-short is finally readable. 2. THE DEPLOY GATE STATES ITS OWN BAR. 'edge -1pp' era after era cannot separate 'short by a hair' from 'short by an amount no strategy could cover'. The era line now prints the required win rate, the SE, the effective n and the lifespan it was deflated by; above 100% it says UNREACHABLE. At 4,738 OOS bars and L=75.6 there are ~63 independent observations, putting the bar near 66% at typical coverage. 3. LIFESPAN MEASURED PER RUNG. The first-passage cache already stores touch ages at every ladder level, so each candidate geometry's resolution time is readable without training on it - L-vs-width becomes a measurement across the whole ladder in ONE run rather than a second chart. Each rung reports L, n_eff, min provable edge and min provable EV. 4. SCALE OBJECTIVE IS PHASE-AWARE, defaulting to MEASURE. Width and detectability are opposed: labels overlap by L, L grows like m*k = width^2 at fixed ratio, so min provable EV ~ width^2 while the cost saving from width is only linear. Doubling width quadruples the smallest EV you can prove. DEPLOY (widest that clears reachability) is right once an edge is known; MEASURE (narrowest that keeps round-trip spread under BARRIER_MAX_COST_FRACTION_PCT) is right while it still has to be shown. The direction does not depend on the exponent, and item 3 makes the exponent checkable. Fixed in review: m_lastRungLifespan is cleared on every LadderWinShare entry or a rejected rung reports the previous rung's lifespan as its own; per-rung detectability is labelled IS-sample based (the deriver may not see the holdout), so absolute figures are optimistic while the ranking is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:37:05 -04:00
//--- DENOMINATOR IS THE PER-ERA BAR COUNT, not m_oosSamples (fixed 2026-08-17). This is the exact
//--- bug already caught and fixed for logBuyPredPct thirty lines up - "era-15 Buy:2% that was
//--- really ~30%" - and it was left sitting in the one line whose entire job is to be the
//--- reference every other number on this line is read against. m_oosWinLongTotal is reset every
//--- era; m_oosSamples only resets on a full model reset, so it accumulates across the whole run
//--- and this ratio decayed as ~1/era. It was therefore correct at era 1 and wrong everywhere
//--- after: an SP500 H4 run whose true always-long rate is 37% printed 1.2% at era 33 and 0.0%
//--- at era 2219, which is what made the always-short figure look like a broken field rather
//--- than a diluted one. Any historical reading of this line is invalid unless it came from
//--- era 1 - including the "62% zero-skill" figure quoted in the 2026-08-16 notes, which was
//--- derived by hand and only ever agreed with this line at the very start of a run.
int zsBars = m_oosBuyTotal + m_oosSellTotal + m_oosNeutralTotal;
string zeroSkillInfo = (zsBars <= 0) ? "" :
fix(batchnorm): bound the normalized value - a constant input feature was amplified 1e4x and pinned PAI's head to its rails BN_MIN_STD = 1e-4 caps the per-unit gain at 1/1e-4 = 1e4, and the comment above it states that as though it were a safety property. It is not. A unit whose running variance is ~0 is a CONSTANT feature carrying no information, and dividing its rounding noise by 1e-4 hands the next layer an activation of several hundred. BN's contract is "output has ~unit variance"; a unit that cannot supply that must contribute nothing, not the largest signal in the layer. MEASURED, 2026-08-17 SP500 H4, four topologies on identical separate charts: model spread Neutral CHOSE Neutral TIED rail CONV 0.386 0.68% 0.10% 0.48% LSTM 0.392 0.63% 0.00% 0.00% HYB 0.376 1.79% 0.00% 0.01% PAI 0.192 0.09% 80.63% 99.99% bn1's cached nx normed 1.38e4 over 800 units. PAI's SIGMOID head was on its rails on 99.99% of bars, with Buy and Sell landing on the SAME rail so they compared exactly equal, and ApplyClassificationSoftmax()'s strict-majority rule reported that tie as Neutral on ~80% of bars. So the long-running "PAI is heavily biased toward Neutral" was never a class-prior problem: the net CHOSE Neutral on 0.09% of bars. It was float equality on a saturated head. The 331ab29 counters answered it on their first run. PAI-only because it is the one topology whose FIRST batch norm sits on the raw 800-dim input vector - CONV/LSTM/CONVLSTM all have a conv or LSTM stage in front, so their first BN sees a learned representation with no degenerate units. That asymmetry was already on file as a suspicion; this is the mechanism. FIX, mirrored in both backends (host NeuronBatchNorm.mqh and device Network.cl): forward nx = clamp(delta/sd, -BN_MAX_NX, +BN_MAX_NX), BN_MAX_NX = 8 backward if the forward bound this unit, the output stopped depending on the input, so d(nx)/dx = 0 and NO gradient passes The backward half is not optional. g is divided by the same sd the forward multiplies by, so a degenerate unit gets its GRADIENT amplified 1e4x too - the "receives gradients divided by sqrt(var) ~ 500" pathology already noted in Network.cl's Adam kernel. Bounding only the forward would move the explosion downstream. 8 sigma is inert on anything healthy (|nx| > 8 is a ~1e-15 event under normality); it binds only on degenerate units, which is the entire point. Same clamp-to-range idiom the activation derivatives beside it already use. SelfCheckBnForward/SelfCheckBnHiddenGrad already prove host against kernel, and BN_OPT_NX was already consumed in the backward for the gamma gradient, so the new read adds no lifetime assumption. Expect PAI to change behaviour and CONV/LSTM/CONVLSTM not to (their rail rate is ~0%, so the clamp never binds). No .nnw format or fingerprint change. ALSO: print the zero-skill reference on the era line. m_oosWinLongTotal and m_oosWinShortTotal have been accumulated for a long time and NEVER printed, which is why three separate topologies all sitting at 62% read as a mysterious coincidence rather than the obvious base rate. It is not a coincidence: with the target (1.70 ATR) nearer than the stop (3.07 ATR), BOTH sides win on 24.5% of bars, so winLong+winShort covers ~124% of them and a no-edge caller collects 62.1% whichever way it calls - against a 64.3% break-even. Derived from this run's own label counts: (11329 - 2776 + 2*2776) / (2*11350) = 62.14%. Every win rate on that line must be read against this, not against 50%. NOT COMPILED - user compiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:21:51 -04:00
StringFormat(" | zero-skill on these bars: always-long %.1f%%, always-short %.1f%%,"
feat(measurement): fix zero-skill denominator, publish the deploy bar, measure lifespan per rung, add a MEASURE scale objective The last run could not have demonstrated an edge either way, and nothing in the log said so. Four changes so it does. 1. THE ZERO-SKILL LINE DIVIDED BY THE WRONG DENOMINATOR. m_oosWinLongTotal resets every era; m_oosSamples only resets on a full model reset. So 'always-long %' decayed as ~1/era: a run whose true rate is 37% printed 1.2% at era 33 and 0.0% at era 2219. This is the SAME bug already found and fixed for logBuyPredPct thirty lines above ('era-15 Buy:2% that was really ~30%'), left in the one line whose whole job is to be the reference every other number is read against. Correct at era 1, wrong everywhere after - including the '62% zero-skill' figure in the 2026-08-16 notes. Now per-era, and always-short is finally readable. 2. THE DEPLOY GATE STATES ITS OWN BAR. 'edge -1pp' era after era cannot separate 'short by a hair' from 'short by an amount no strategy could cover'. The era line now prints the required win rate, the SE, the effective n and the lifespan it was deflated by; above 100% it says UNREACHABLE. At 4,738 OOS bars and L=75.6 there are ~63 independent observations, putting the bar near 66% at typical coverage. 3. LIFESPAN MEASURED PER RUNG. The first-passage cache already stores touch ages at every ladder level, so each candidate geometry's resolution time is readable without training on it - L-vs-width becomes a measurement across the whole ladder in ONE run rather than a second chart. Each rung reports L, n_eff, min provable edge and min provable EV. 4. SCALE OBJECTIVE IS PHASE-AWARE, defaulting to MEASURE. Width and detectability are opposed: labels overlap by L, L grows like m*k = width^2 at fixed ratio, so min provable EV ~ width^2 while the cost saving from width is only linear. Doubling width quadruples the smallest EV you can prove. DEPLOY (widest that clears reachability) is right once an edge is known; MEASURE (narrowest that keeps round-trip spread under BARRIER_MAX_COST_FRACTION_PCT) is right while it still has to be shown. The direction does not depend on the exponent, and item 3 makes the exponent checkable. Fixed in review: m_lastRungLifespan is cleared on every LadderWinShare entry or a rejected rung reports the previous rung's lifespan as its own; per-rung detectability is labelled IS-sample based (the deriver may not see the holdout), so absolute figures are optimistic while the ranking is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:37:05 -04:00
" coin-flip %.1f%% (the gate ranks on the LARGER of the first two; the gap"
" between them IS the directional drift, and a model that only reproduces it"
" has found the drift, not an edge) | break-even %.1f%% frictionless, %.1f%%"
" AFTER SPREAD (%.3f*ATR)",
100.0 * (double)m_oosWinLongTotal / zsBars,
100.0 * (double)m_oosWinShortTotal / zsBars,
50.0 * ((double)m_oosWinLongTotal + (double)m_oosWinShortTotal) / zsBars,
fix(geometry): the target was small BY CONSTRUCTION - ratio is now policy, scale is measured, ladder ceiling removed The derivation read the stop from q75 of ADVERSE travel and the target from q50 of FAVOURABLE travel. Over one horizon those distributions are broadly the same shape, so q75 > q50 MECHANICALLY - the target came out smaller than the stop no matter what the market did. SP500 H4 shipped stop 3.07 / target 1.70: a 0.55:1 payoff needing 64.3%. That was never a measurement, it was two mismatched constants. The reachability line printed beside it - "target on 50.0% of bars, stop on 25.0%" - is exactly 1-q50 and 1-q75. Tautological. It cannot disconfirm anything, and it read as validation. WIDTH AND RATIO ARE INDEPENDENT AND ONLY ONE PAYS. EV = edge x width; ratio is EV-neutral (a driftless walk reaches +m before -k with probability k/(k+m), which IS break-even). Width is what buys cost efficiency: the spread is a fixed 0.047*ATR here, so the shipped 4.77*ATR width paid it 21 times per unit of travel. So: RATIO = policy. BARRIER_TARGET_RR = 2.0 (user's 1:2). Break-even 33.3%. SCALE = measured. The stop quantile is chosen from a ladder, WIDEST FIRST, taking the first rung whose implied 2x target is still reached often enough to be a trainable class. That last clause is the difference from the min-reward:risk raise removed in 2026-08-09, which forced target = 2 x stop with NO reachability test, landed on 6.66*ATR reachable on 3.3% of bars, and trained the model to predict something that essentially never happened. Same ratio; the scale now retreats until the data says the target is attainable. Every rung is logged. LADDER CEILING REMOVED. BARRIER_LADDER stopped at 5.00 and the expectancy scan's "best resolvable pair on width alone" came back as stop 5.05 / target 4.95 - it pinned to the top rung. A recommendation landing exactly on the edge of its own search space is a boundary, not a finding: it cannot tell "5 ATR is optimal" from "5 ATR is all we allowed". Extended to 20*ATR (8 -> 14 rungs). Nothing else needs editing - every consumer is parameterised by BARRIER_LADDER_COUNT - and the horizon constraints (decided >= 60%, reachability floor) now bind instead of a constant. THE SCAN COULD NOT SEE THE SHIPPED GEOMETRY. ReportBarrierGeometryScan looked the configured pair up in its integer grid, and DeriveBarrierGeometry produces CONTINUOUS multiples (3.07/1.70) that can never equal a grid point - so cfgExcess stayed at its -1.0 sentinel and the report printed "configured 3:2 scores -1.00000", which reads as a catastrophic score and actually means "never evaluated". Worse, the grid skipped target<stop entirely because it "inverts the trade's whole premise" - while the derivation was shipping exactly that. The incumbent is now always scored as a peer (never crowned; it is already in force and is not an enum pairing the scan could adopt). BREAK-EVEN NOW INCLUDES THE SPREAD. Every report quoted the frictionless SL/(SL+TP). On SP500 H4 that read 64.3% while the MEASURED zero-skill rate was 62.1% - a 2.2pp gap that IS the cost, and that made every model look 2.2pp better than it was. CostAdjustedBreakEvenPct() prices a win at (TP - spread) and a loss at (SL + spread), matching the expectancy scan's convention exactly so the two reports cannot disagree. It also feeds FitDirConfThreshold, which is the correctness half: the operating point subtracts break-even from precision, so the frictionless figure made every candidate threshold look better by the width of the spread - 2.2pp against a measured edge of 2.3pp, i.e. very nearly all of it. Era line now carries both: "break-even 64.3% frictionless, 66.6% AFTER SPREAD". Forces a full relabel and retrain. Requested. NOT COMPILED - user compiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:34:32 -04:00
beFrictionless, CostAdjustedBreakEvenPct(), m_spreadAtr);
feat(measurement): fix zero-skill denominator, publish the deploy bar, measure lifespan per rung, add a MEASURE scale objective The last run could not have demonstrated an edge either way, and nothing in the log said so. Four changes so it does. 1. THE ZERO-SKILL LINE DIVIDED BY THE WRONG DENOMINATOR. m_oosWinLongTotal resets every era; m_oosSamples only resets on a full model reset. So 'always-long %' decayed as ~1/era: a run whose true rate is 37% printed 1.2% at era 33 and 0.0% at era 2219. This is the SAME bug already found and fixed for logBuyPredPct thirty lines above ('era-15 Buy:2% that was really ~30%'), left in the one line whose whole job is to be the reference every other number is read against. Correct at era 1, wrong everywhere after - including the '62% zero-skill' figure in the 2026-08-16 notes. Now per-era, and always-short is finally readable. 2. THE DEPLOY GATE STATES ITS OWN BAR. 'edge -1pp' era after era cannot separate 'short by a hair' from 'short by an amount no strategy could cover'. The era line now prints the required win rate, the SE, the effective n and the lifespan it was deflated by; above 100% it says UNREACHABLE. At 4,738 OOS bars and L=75.6 there are ~63 independent observations, putting the bar near 66% at typical coverage. 3. LIFESPAN MEASURED PER RUNG. The first-passage cache already stores touch ages at every ladder level, so each candidate geometry's resolution time is readable without training on it - L-vs-width becomes a measurement across the whole ladder in ONE run rather than a second chart. Each rung reports L, n_eff, min provable edge and min provable EV. 4. SCALE OBJECTIVE IS PHASE-AWARE, defaulting to MEASURE. Width and detectability are opposed: labels overlap by L, L grows like m*k = width^2 at fixed ratio, so min provable EV ~ width^2 while the cost saving from width is only linear. Doubling width quadruples the smallest EV you can prove. DEPLOY (widest that clears reachability) is right once an edge is known; MEASURE (narrowest that keeps round-trip spread under BARRIER_MAX_COST_FRACTION_PCT) is right while it still has to be shown. The direction does not depend on the exponent, and item 3 makes the exponent checkable. Fixed in review: m_lastRungLifespan is cleared on every LadderWinShare entry or a rejected rung reports the previous rung's lifespan as its own; per-rung detectability is labelled IS-sample based (the deriver may not see the holdout), so absolute figures are optimistic while the ranking is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:37:05 -04:00
//--- THE DEPLOY BAR, stated. Reading "edge -1pp" era after era tells you the model is short; it
//--- does not tell you whether it is short by a hair or by an amount no strategy could ever
//--- cover. That distinction is the difference between "keep training" and "the measurement
//--- design is wrong", and it is the single most expensive thing this log could not say.
string gateInfo = (m_lastEdgeFloorPct < 0.0 || m_lastEffN <= 0.0) ? "" :
StringFormat(" | DEPLOY BAR %.1f%% (chance + %.0f x SE %.1fpp on %.0f INDEPENDENT calls -"
" %d raw calls deflated by the %.1f-bar mean label lifespan)%s",
m_lastEdgeFloorPct, EDGE_MIN_SIGMAS, m_lastPrecSE, m_lastEffN,
(int)(m_oosBuyFired + m_oosSellFired), MeanLabelLifespan(),
//--- A bar above 100% is not "hard", it is unreachable, and no amount of
//--- training addresses it - only a bigger independent sample does.
(m_lastEdgeFloorPct >= 100.0
? " <-- UNREACHABLE: no win rate can clear this. The OOS window does not hold"
" enough independent observations to certify ANY edge; widen the sample"
" (more instruments / lower timeframe) or narrow the barrier."
: ""));
feat(diagnostics): split a reported "Neutral" into CHOSE vs TIED - they need opposite fixes ApplyClassificationSoftmax() requires a STRICT majority over both rivals and sends every tie, 2-way or 3-way, to Neutral. So "OOS recall Neutral:100%" is two completely different events sharing one label: CHOSE - the net genuinely ranks Neutral highest. A class-prior/label problem. TIED - the top two are EXACTLY equal, so the net expressed no preference and the tie-break reported Neutral. A SATURATION problem: the head is SIGMOID, and a saturated sigmoid returns exactly 0.0f or 1.0f in the DLL's float32, so two classes pinned to the same rail compare equal and the bar is silently discarded. Nothing in the logs could tell them apart, and the fixes point opposite ways. Eras 1-25 of the 2026-08-17 solo PAI run read "Neutral 100%" at spread avg 0.99 - fully saturated - and broke out at era 27 as the spread fell to 0.75. That is consistent with EITHER story. The user reports the Neutral phase on most runs, so it is worth four longs to stop guessing. Four per-era counters on the pass 3 OOS walk, reported as: | Neutral CHOSE 12.4% / TIED 38.1% (of which B=S 1204) | rail 61.2% m_oosNeutralStrict - Neutral strictly highest m_oosNeutralTie - no strict winner; the tie-break produced Neutral m_oosTieBuySell - the costly subset: Buy and Sell tied AT the top, i.e. a DIRECTIONAL reading thrown away by float equality m_oosRailBars - any raw output sitting on a sigmoid asymptote, the saturation that makes exact ties possible at all Read on the RAW logits, before ApplyClassificationSoftmax() overwrites TempData in place. Legitimate because softmax is strictly monotone: it cannot change the ordering and cannot break a tie either, so the raw reading and the decision always agree. Placed alongside the existing min/max/spread capture so all the output diagnostics describe the same values. Measurement only - no decision path reads these. NOT COMPILED - user compiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:06:54 -04:00
string neutralWhy = (m_oosOutCount <= 0) ? "" :
StringFormat(" | Neutral CHOSE %.1f%% / TIED %.1f%% (of which B=S %d) | rail %.1f%%",
100.0 * (double)m_oosNeutralStrict / m_oosOutCount,
100.0 * (double)m_oosNeutralTie / m_oosOutCount,
(int)m_oosTieBuySell,
100.0 * (double)m_oosRailBars / m_oosOutCount);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- No "(target X%)" any more - there is no absolute accuracy target. What replaces it as the
//--- progress indicator is the plateau counter: how many eras since the last new best, and how
//--- close that is to ending the run (see the PLATEAU_* ladder).
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
string plateauInfo = (m_bestBalancedOos < 0) ? "" :
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
(" | best bal " + DoubleToString(m_bestBalancedOos, 1) + "%, " + IntegerToString(m_erasSinceBestBalanced) +
refactor(ai): nine class-imbalance inputs down to two The imbalance section offered nine controls for one job. Audited against the code, five of them did not do what their names said at the shipped defaults: AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns whenever the adjusted loss is on, which is default. OversampleParity DEAD in training - Training.mqh gated the replay loop on !useLogitAdjustedLoss (correctly, citing Buda et al. 2018). Live only in the online-learning path. EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma damper - "replay minority bars through pass-2 oversampling" was a focal-loss switch. ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25. UseStaticPrior An exact duplicate of FreezePriorCalibration - the two were OR'd together in the single place either is read. So they were not five mechanisms fighting; they were one mechanism plus eight knobs that mostly described machinery that no longer ran. That is worse than a real conflict, because the log agreed with the names: the label-cache line printed "reps up to 28x (90% parity) (seeding era 0's class-balance oversampling)" on every run, describing an oversampling pass that had been switched off. It is fixed here too - it cost this session a wrong diagnosis. The one genuine redundancy was focal loss, running at gamma*0.125 alongside the adjusted loss: two corrections on the same axis, the exact stacking failure this file already cited Buda et al. for in two other places, damped by a replay flag whose replay path was itself dead. Removed rather than re-tuned. The plateau ladder is unaffected - its escape is the learning-rate warm restart; the gamma anneal beside it only ever stepped toward zero. WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze: LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted- Loss boolean, since a strength dial where 0 already means off does not need an on/off switch beside it. FreezePriorCalibration unchanged. It is the only one of the six corrections with a consistency guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection already ranks on - so the loss and the deploy decision optimize one thing. The online continual-learning path keeps its own alpha-balanced focal weight, now as constants pinned to the removed inputs' shipped defaults, so its behaviour is unchanged. It legitimately needs its own correction: ApplyLogitAdjustment() only runs inside a training run, so a deployed model that was reloaded carries no logit offsets and would otherwise stream 31:1 data into itself uncorrected. The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a double fed to a %d conversion and had always emitted a literal 0; the |MR: segment is written as the constant its shipped defaults produced. Dropping either would have re-keyed every model and forced a from-scratch retrain of the one topology currently converged and trading. Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable "neutralized by prior correction" diagnostic. Both builds compile 0 errors, 0 warnings. No retrain forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
" eras since (stage " + IntegerToString(m_plateauStage) + "/" + IntegerToString(PLATEAU_STAGE_DEPLOY) + ")");
fix(ui): unique chart tag, product-grade panel, responsive under load Three separate reports from one deploy. 1. CONV, LSTM and HYBRID all came back tagged [4109]. The weights fingerprint omits the topology type on purpose - the file path already separates it (State\CONV\ vs State\LSTM\ vs State\HYB\) and hashing a value that is constant within a folder buys nothing while re-keying every trained model into a forced retrain. So the files were never at risk, but the tag could not do its one job. Prefixing the short id makes it unique on the display side only; the hex half still greps straight to the .nnw inside the folder the prefix names. 2. The default panel read like a training console. Six lines down to three, each answering a question an owner actually has. The deploy internals (best score, eras-since-best, ladder stage) were developer diagnostics describing a recall floor that no longer decides anything, and were already in the era-end journal line. In-sample accuracy left the panel too: it grades the model on bars it trained on, so it always flatters, and showing it beside the honest number invites reading the wrong one. New compile-time DebuggingMode constant - deliberately not an input - carries the IS/OOS pair and the resolved model path into the journal instead. No extra Inputs row, no extra Market description line, no user-reachable firehose. 3. Panel drag and buttons stuttered under training load, exactly as the 2026-07-26 note raising the chunk budget to 200ms warned they might. Backed off to the documented 120ms - worst-case click latency is that budget - and the derived topology (~292k weights to ~29k) makes the throughput this costs far cheaper than when that note was written. Also halved the panel redraw rate to 2.5 Hz: ChartRedraw repaints the whole chart, so its cost scales with accumulated arrows, and 5 Hz was the larger half of the stutter. Era-end still force-refreshes. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 09:05:58 -04:00
//--- Lifetime IS/OOS directional accuracy. The panel now shows the out-of-sample half alone (see
//--- ComputeCompoundedAccuracyLine - the in-sample figure grades the model on bars it trained on,
//--- so it always reads higher than anything forward trading will deliver and does not belong on
//--- a product's face). The GAP between the two is still the over-fitting read, so it survives
//--- here, once per era, behind the compile-time DebuggingMode constant.
string lifetimeInfo = (!DebuggingMode || (m_cumIsTotal <= 0 && m_cumOosTotal <= 0)) ? "" :
(" | lifetime dir acc IS " + (m_cumIsTotal > 0 ? IntegerToString((int)MathRound(m_cumIsCorrect * 100.0 / m_cumIsTotal)) + "%" : "n/a") +
" OOS " + (m_cumOosTotal > 0 ? IntegerToString((int)MathRound(m_cumOosCorrect * 100.0 / m_cumOosTotal)) + "%" : "n/a") +
" over " + IntegerToString(m_cumIsTotal + m_cumOosTotal) + " calls");
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
//--- Wall-clock split for the era that just finished, but only when it was SLOW - a healthy
//--- era stays exactly one line. Same purpose as TrainHeartbeat, for the completed case: an
//--- era that took 30 minutes must say where the minutes went, or it is undiagnosable from
//--- the outside (2026-08-10).
fix: prebuild and era sized different windows; diag: Train() names its branch TWO things, one incident. 1) THE BUG I SHIPPED IN 0c85c54. m_tuneStartTrainBar is declared, initialised to 0, and NEVER ASSIGNED - the assignment existed before the God-class split and the split dropped it, leaving a dead member. Harmless while nothing read it; a real defect the moment 0c85c54 made StartLabelCachePrebuild() reset dtStudied from it. Train() then computed the window as max(StartTrainBar, floor) while the prebuild computed max(0, floor), where StartTrainBar is the non-zero datetime OnChartEventHandler passes through from the "New Bar" event. The two therefore disagreed about `bars`, so EnsureBarCachesCapacity() saw a changed size at era start, wiped the caches, and re-armed a full 38k-bar prebuild - instead of training. Restored the assignment so both sides evaluate the identical expression. 2) THE REASON IT TOOK ALL NIGHT TO FIND. Train() is a state machine with six early-return branches above the era loop and every one of them is silent. Four charts burned a core each for 15 minutes with an empty journal: the pass heartbeats (694b756) proved the era loop was never reached, no prebuild completion line appeared either, and nothing external can see inside a single MQL5 thread - per-thread CPU says "busy", file writes say nothing, and the VPS has no debugger. That is an undiagnosable state, and it is the thing to fix, not just the bug of the day. ReportTrainStall() now names the branch Train() is taking whenever no era has completed for 3 minutes, at most once a minute per signal, with the state that decides the branch: run/prebuild/simOos/resume flags, era, dtStudied, and - for the cache-invalidation branch specifically - BOTH bar counts, since two sizings disagreeing is exactly what re-arms the prebuild forever. Silent on a healthy run: an era completing resets the clock. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:32:08 -04:00
//--- An era finished: the stall clock restarts from here (see m_lastEraCompleteTick).
m_lastEraCompleteTick = GetTickCount();
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
string eraTimeInfo = "";
{
double eraS = (GetTickCount() - m_eraStartTick) / 1000.0;
if(eraS > 120.0)
perf: the excursion head cost 3.6x era time - cut its dispatches ~250x Measured on exc-race-v3: LSTM era 300s -> 1087s (net 272->748s, "other" 30->337s). My estimate had been "single-digit percent". The cost is per-DISPATCH, not per-FLOP, and therefore hits EVERY backend: the head is 19k weights and ~2.4 GFLOP an era - seconds of arithmetic - but ~48k forward/backward calls x several layer submits each, and its 760-wide layer exceeds the CPU DLL's inline threshold so each one pays a real handoff. The classifier's own net time tripled too, from contention with a second pool on an already-full box. Three changes, all backend-neutral because they remove submits rather than tune threads: SCORE ONLY DISJOINT WINDOWS (~64x). Adjacent bars share all but one bar of their horizon, so 16k consecutive bars were always ~250 independent observations - the full-sample tally was never worth more than the disjoint one, it just quoted an n that was ~64x too large. Dropping it costs nothing statistically and removes 63 of every 64 forward passes. The two parallel tallies collapse into one, which is also less code. The trailing ring still advances on every bar: it needs the outcome SEQUENCE, and that is array lookups, not a forward pass. TRAIN ON EVERY 4th PRIMARY BAR (4x). The target is low-dimensional and strongly autocorrelated - neighbouring bars carry near-identical excursion information - so per-bar training buys resolution the target does not have. Strided on ATTEMPTS, not acceptances, so a stretch of unlabelled bars cannot silently change the spacing. OWN TIMING COLUMN. The head's passes were landing in the era line's "other" bucket, which is how a 3.6x regression read as an unexplained jump in the one column nobody attributes. A cost that cannot be seen in the timing line cannot be traded off against anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:38:29 -04:00
//--- The excursion head gets its OWN column. It used to fall into "other", where a 3.6x
//--- era-time regression showed up as an unexplained jump in the one bucket nobody
//--- attributes - a cost invisible in the timing line cannot be traded off against
//--- anything. "other" is now genuinely everything else.
eraTimeInfo = StringFormat(" | ERA TOOK %.0fs (feature windows %.0fs, net fwd/back %.0fs,"
" excursion head %.0fs, other %.0fs)",
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
eraS, m_passFeatUs / 1000000.0, m_passNetUs / 1000000.0,
perf: the excursion head cost 3.6x era time - cut its dispatches ~250x Measured on exc-race-v3: LSTM era 300s -> 1087s (net 272->748s, "other" 30->337s). My estimate had been "single-digit percent". The cost is per-DISPATCH, not per-FLOP, and therefore hits EVERY backend: the head is 19k weights and ~2.4 GFLOP an era - seconds of arithmetic - but ~48k forward/backward calls x several layer submits each, and its 760-wide layer exceeds the CPU DLL's inline threshold so each one pays a real handoff. The classifier's own net time tripled too, from contention with a second pool on an already-full box. Three changes, all backend-neutral because they remove submits rather than tune threads: SCORE ONLY DISJOINT WINDOWS (~64x). Adjacent bars share all but one bar of their horizon, so 16k consecutive bars were always ~250 independent observations - the full-sample tally was never worth more than the disjoint one, it just quoted an n that was ~64x too large. Dropping it costs nothing statistically and removes 63 of every 64 forward passes. The two parallel tallies collapse into one, which is also less code. The trailing ring still advances on every bar: it needs the outcome SEQUENCE, and that is array lookups, not a forward pass. TRAIN ON EVERY 4th PRIMARY BAR (4x). The target is low-dimensional and strongly autocorrelated - neighbouring bars carry near-identical excursion information - so per-bar training buys resolution the target does not have. Strided on ATTEMPTS, not acceptances, so a stretch of unlabelled bars cannot silently change the spacing. OWN TIMING COLUMN. The head's passes were landing in the era line's "other" bucket, which is how a 3.6x regression read as an unexplained jump in the one column nobody attributes. A cost that cannot be seen in the timing line cannot be traded off against anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:38:29 -04:00
m_excUs / 1000000.0,
MathMax(eraS - m_passFeatUs / 1000000.0 - m_passNetUs / 1000000.0
- m_excUs / 1000000.0, 0.0));
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
}
feat(logs): throttle the settled per-era diagnostics - measured 22MB/9.5h of confirmed-working systems Measured from the journal (2026-08-19): the era deep-dive line (~2KB) plus the excursion verdict, tier re-rank, calibration move, barrier hold and selection-regressed note each printed EVERY era for EVERY member - ~940 eras/member/day - long after the systems they watch were confirmed working. Yesterday's file was 1.3GB (70% of it the news-filter calendar spam the sweep fix already removed). VerboseMode returns as an INPUT (demoted 2026-08-01 for the marketplace; that track is dead since the 2026-08-16 pivot) and gains a second job: false throttles each settled per-era print to eras 0-3 plus every TRAIN_LOG_EVERY_ERAS-th (25 ~= one deep-dive per ~15min per member); true restores the per-era firehose, flippable live. Never throttled: anything that marks a CHANGE - new bests, restores + eta decays, plateau stage transitions, deploy approvals, warnings, errors, the label-cache/adoption one-shots, and the combined-vote gate line (the active system's primary telemetry, still every era). Semantic fixes over blanket gating: - barrier hold now ARMS silently and prints only when the hold outlasts the 2-min report interval - a brief hold every era is the design, the long hold is the watchdog case the line exists for; - the ensemble deploy REFUSAL prints immediately when its reason changes (that is a finding), on cadence when unchanged; - the filtered-view census prints when its RESULT moves (drawn count, or strongest vote by >=2pp) and at least every 10th sweep. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 09:36:15 -04:00
//--- THROTTLED (2026-08-19): this is the ~2KB deep-dive block, and it printed every era
//--- for every member - ~3.7MB per member per day, the single largest line in a measured
//--- 22MB/9.5h journal. The panel still refreshes every era (below), the combined-vote
//--- gate line still prints every era, and the member HUD shows the live numbers; this
//--- full block keeps the TRAIN_LOG_EVERY_ERAS cadence so a run stays reconstructable
//--- from the file without drowning the console. VerboseMode = every era again.
if(TrainLogDue())
Print(ID + ": training in progress - era " + IntegerToString(m_eraCount) + ", OOS accuracy " + DoubleToString(dOosForecast, 1) + "%, IS error " + DoubleToString(dError, 2) + recallInfo + balancedInfo + selectionInfo + predictedInfo + liveInfo + tierInfo + plateauInfo + lifetimeInfo + zeroSkillInfo + gateInfo + m_lastPoolReport + rawOutInfo + neutralWhy + layerInfo + eraTimeInfo);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
// Forced (unthrottled) panel refresh, right here alongside the console line above, using this
// era's own just-finalized m_eraCount/dOosForecast - see UpdateTrainingStatusLabel's
// declaration comment for why this can't just rely on the next throttled bar-scan call to
// catch up (it would, but a full era later than the console already reported it).
UpdateTrainingStatusLabel("Era complete", m_lastDisplayNeuron0, m_lastDisplayNeuron1, m_lastDisplayNeuron2, m_lastDisplaySignal, true);
}
//--- Genuine convergence THIS era (not a stale m_trainingComplete carried over from a previous
//--- run) - (re)start the evaluation-only continual-learning OOS walk. Always rebuilt fresh from
//--- the just-converged weights; never resumes a stale walk from a superseded model.
//--- m_trainingComplete is the plateau ladder's verdict now (see where it is assigned): "stopped
//--- improving after both escape attempts, and there is a recall-passing checkpoint to deploy".
//--- It replaces the old (m_objectiveMet && m_oosStable) test, which depended on the removed
//--- absolute accuracy target to mean anything - without it, m_oosStable alone (3 eras inside a 2pp
//--- band) would have declared convergence at the first flat spot in every run.
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
if(!stop && m_trainingComplete)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
{
feat(ai): measure precision per confidence tier; fix stale metric labels Two things the 2026-07-30 run exposed. 1. Every user-facing message still called the selection metric "balanced accuracy". It has ranked on directional precision since a142749, so "CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5% PRECISION as if it were macro-recall, while the same era logged an actual balanced accuracy of 49%. Two different numbers under one name, in the line that announces a deploy. Relabelled at every site, including the stage-3 refusal, which still described the per-class recall floor that stopped being the gate. 2. Precision is now bucketed by confidence tier and logged per era, both per-tier and cumulatively from each tier upward: | tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ... The per-tier number says whether confidence is calibrated to correctness at all; if it does not rise T0->T3, raising the floor buys nothing and that is the finding. The ">=" number is what a floor would actually deliver, with its fire count, so the coverage cost is visible in the same line. Tier weights are 25/50/75/100, so for an AI-only config Min_Vote_Open maps straight across: 50 = ">=T1", 75 = ">=T2", 100 = ">=T3". Bucketing happens at the existing live-fired accounting site, so it measures exactly the population that trades - not the raw argmax. Both builds compile 0 errors, 0 warnings. No retrain needed for either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
Print(ID + ": training CONVERGED at era " + IntegerToString(m_eraCount) + " - this is the best this configuration reached: dir-precision " +
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
DoubleToString(m_bestBalancedOos, 1) + "%, blended OOS " + DoubleToString(dOosForecast, 1) + "%, IS error " + DoubleToString(dError, 2) +
diag: report |dW| alongside d|W| per layer, and drop two stale log claims The per-era `dW/W` line measured the change in each layer's weight NORM. That statistic cannot separate "this layer only shrank under weight decay" from "this layer moved somewhere useful" - a rotation at constant norm and pure decay can print the same number. It matters right now: on SP500 H1 the LSTM layers print a near-constant ~1.05%/era that exactly equals their geometric norm decay over 318 eras (HYB lstm2 12.966 -> 0.755, monotone, never once up), while a sibling conv oscillates around a much slower drift. Norm-change can only hint at that. Now prints norm(d|W|% / |dW|%). Under decay alone the two are equal; any gradient component adds in quadrature to the second, so a learning layer shows the second clearly larger. Diagnostic only - no training behaviour changes, fingerprint untouched. Also removes two log lines that described machinery deleted in 397b0ea: the plateau ladder's terminal stage still claimed "after a warm restart AND full gamma anneal", and the CONVERGED line "across a warm restart and a full focal-gamma anneal". Both printed on every CONV/LSTM convergence today. Same failure mode as the oversampling line 397b0ea fixed: a log that describes what an older version would have done is confirming evidence for a false hypothesis. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 14:33:29 -04:00
". No new best for " + IntegerToString(m_erasSinceBestBalanced) + " eras across " +
IntegerToString(PLATEAU_STAGE_DEPLOY - 1) + " learning-rate warm restarts." +
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
" Weights saved, switching to live inference.");
StartOosContinualSimulation(bars, oosCutoff);
}
if(stop || m_trainingComplete)
FinalizeTrainRun();
feat: implement one-shot pattern-database backfill and enhance accuracy tracking for ensemble models
2026-08-16 21:08:41 -04:00
//--- Deliberately AFTER FinalizeTrainRun(): that call restores the DEPLOYED checkpoint's weights
//--- (which may differ from the last era's, if the plateau ladder's best era wasn't the last one
//--- run), and this backfill must score with exactly what is about to trade live.
if(!stop && m_trainingComplete)
fix: drop the ranking slice for the calibration band; un-collapse the tiers NOT COMPILED - user compiles. (1) THE RANKING SLICE IS GONE. It reserved 20% of the OOS window so the pattern-DB backfill would read bars the deployed checkpoint was not SELECTED on. That objection stands; carving a new region to answer it did not. The calibration band already has every property the slice was buying: never trained on | never graded by pass 3 (which walks [0, oosCutoff) and so never reaches it) | never seen by the deploy gate | purged by a full label horizon on BOTH sides | and larger besides - 1,684 bars vs the ~970 carved So the backfill now walks [calibLo, calibHi) and pass 3 goes back to grading the entire OOS window, exactly as before any of this. The gate gets its full sample back (~10% of a sigma), the split loses a region, and the failure mode found an hour ago - a reserved region silently blanking ~10 months of chart arrows, because arrows are only drawn on bars pass 3 grades - becomes impossible. One impurity, stated in the completion log rather than hidden: m_dirConfThreshold is FITTED on that band and the walk applies it to decide which bars fired, so coverage there is mildly optimistic. One scalar under a coverage floor, against checkpoint selection over hundreds of eras. This backfill IS the deploy-time warm-up: it runs right after FinalizeTrainRun() restores the deployed weights, so it scores with exactly what is about to trade. (2) EVERY CALL WAS TIER 0, AND IT WAS ARITHMETIC. ConfidenceTier() quartiles [floorConf, 1] where floorConf = 1/3 - the lowest magnitude a 3-way softmax winner can hold. But it was fed CalibratedConfidenceMagnitude(), which multiplies by m_confidenceCalScale, clamped to [0.3, 1.5]. That lower clamp is BELOW 1/3. Whenever calibration bottoms out, t goes negative and MathMax(0, ...) pins every call to tier 0. Which is what the live run does. m_confidenceCalScale is EMA'd toward empiricalAccuracy / avgClaimedConfidence; with the model over-calling Neutral, 3-class agreement sits near 10% against a claimed confidence near 0.9, so the ratio is ~0.11 and clamps to 0.3 every era. Logged: tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0) 828 calls, one bucket - the four tier weights and the entire per-tier pattern-DB ranking reduced to a single number. The backfill was feeding a mechanism that structurally could not rank. Tiering now reads the RAW head magnitude, which genuinely lives on the [1/3, 1] range these bounds were written for. Calibration keeps its real jobs - AIConfidence() for MM sizing and SignedAIConfidence() for the vote are unchanged. STILL OPEN, deliberately not touched here: the calibration TARGET itself. empiricalAccuracy is 3-class agreement, which is the wrong quantity to scale a DIRECTIONAL confidence against - it counts a Neutral class that is 0.19% of labels. The honest target is the win rate on the calls the confidence describes (directional precision), with the claimed-confidence average taken over those same called bars. That needs a new accumulator and it interacts with the Neutral over-calling being fixed elsewhere, so it wants one clean run first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 23:06:11 -04:00
StartPatternDatabaseBackfill(bars, totalIter, oosCutoff);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- else: this era is done but the run continues - the next Train() call (re-triggered via
//--- ScheduleTrainingIfNeeded()'s custom event, same mechanism as always) starts the next era
//--- fresh, since m_eraResumePending is false while m_trainRunActive stays true
//--- Save this model's own learning-rate trajectory back out of the shared global before
//--- returning - see m_modelEta's declaration comment. Covers every path that reaches here
//--- (natural era completion, whether or not the run itself just finalized).
m_modelEta = eta;
}
//+------------------------------------------------------------------+
//| Ends the current Train() run: restores the best-scoring era's |
//| checkpointed weights (if any beat the era the loop happened to |
//| end on), persists final state, and clears the resumable-run |
//| flags. Called both from Train() itself (natural stop/converge) |
//| and from StopTraining() (a mid-chunk Stop click won't get |
//| another "New Bar" event to resume into, since |
//| ScheduleTrainingIfNeeded() refuses to schedule while |
//| m_trainingStopRequested is set, so it must finalize synchronously |
//| there instead of being left dangling). |
//+------------------------------------------------------------------+
//| Era-cap decision: keep training (true) or deploy best + stop |
//| (false). Live chart -> operator dialog; headless -> stop. |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::PromptContinuePastEraCap(double bestOos)
{
//--- No GUI in the Strategy Tester/optimizer - MessageBox() is unavailable there and would just
//--- stall a headless run, so deploy the best checkpoint found so far and stop (the safe default).
if(MQLInfoInteger(MQL_TESTER) || MQLInfoInteger(MQL_OPTIMIZATION) || MQLInfoInteger(MQL_FORWARD))
return false;
//--- Reaching this cap is now the UNUSUAL outcome: a run normally ends itself when the plateau ladder
//--- runs out of escapes (see the PLATEAU_* constants), which is a statement about the run having
//--- stopped improving rather than about any accuracy number. So the interesting question here is why
//--- the ladder had not finished yet - either the run was still finding new bests (just needs more
//--- eras), or nothing has ever cleared the per-class recall floor, which blocks auto-deploy on
//--- purpose so a one-class model can never ship. Spell out which.
bool recallMet = (m_lastBuyRecallPct < 0 || m_lastBuyRecallPct >= m_minDirectionalRecallPct) &&
(m_lastSellRecallPct < 0 || m_lastSellRecallPct >= m_minDirectionalRecallPct);
string neutralNote = (m_priorNeutral > 0.0)
? ("inflated by the ~" + IntegerToString((int)MathRound(m_priorNeutral * 100.0)) + "% Neutral base rate")
: "inflated by the dominant Neutral class";
string reasons = "";
if(!m_bestPassedRecall)
reasons += " - No era has ever cleared the per-class recall floor, so there is no model safe to\n" +
" auto-deploy yet (a model that ignores Buy or Sell must never ship)\n";
else
reasons += " - Still improving: " + IntegerToString(m_erasSinceBestBalanced) + " eras since the last new best, plateau stage " +
IntegerToString(m_plateauStage) + " of " + IntegerToString(PLATEAU_STAGE_DEPLOY) + " (the run ends itself at stage " +
IntegerToString(PLATEAU_STAGE_DEPLOY) + ")\n";
if(!recallMet)
reasons += " - Latest era's per-class recall below the floor: Buy " +
(m_lastBuyRecallPct < 0 ? "n/a" : IntegerToString(m_lastBuyRecallPct) + "%") + " / Sell " +
(m_lastSellRecallPct < 0 ? "n/a" : IntegerToString(m_lastSellRecallPct) + "%") +
" (need >=" + IntegerToString(m_minDirectionalRecallPct) + "% each)\n";
if(!m_objectiveMet)
reasons += " - The latest era did not produce a valid model (recall floor not met/not measured)\n";
string balancedStr = (m_bestBalancedOos > 0.0)
feat(ai): measure precision per confidence tier; fix stale metric labels Two things the 2026-07-30 run exposed. 1. Every user-facing message still called the selection metric "balanced accuracy". It has ranked on directional precision since a142749, so "CONVERGED ... balanced accuracy 32.5%" was reporting a 32.5% PRECISION as if it were macro-recall, while the same era logged an actual balanced accuracy of 49%. Two different numbers under one name, in the line that announces a deploy. Relabelled at every site, including the stage-3 refusal, which still described the per-class recall floor that stopped being the gate. 2. Precision is now bucketed by confidence tier and logged per era, both per-tier and cumulatively from each tier upward: | tier prec T0:19%(410)[>=28%/1204] T1:31%(520)[>=34%/794] ... The per-tier number says whether confidence is calibrated to correctness at all; if it does not rise T0->T3, raising the floor buys nothing and that is the finding. The ">=" number is what a floor would actually deliver, with its fire count, so the coverage cost is visible in the same line. Tier weights are 25/50/75/100, so for an AI-only config Min_Vote_Open maps straight across: 50 = ">=T1", 75 = ">=T2", 100 = ">=T3". Bucketing happens at the existing live-fired accounting site, so it measures exactly the population that trades - not the raw argmax. Both builds compile 0 errors, 0 warnings. No retrain needed for either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 11:47:15 -04:00
? ("\nBest directional precision, coverage-weighted (the metric the deployed\ncheckpoint is chosen on): " + DoubleToString(m_bestBalancedOos, 1) +
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
"%\nBest blended OOS accuracy: " + DoubleToString(bestOos, 1) + "% (" + neutralNote + ")\n")
: "";
string msg = ID + ": training reached the " + IntegerToString(m_maxErasPerRun) +
"-era cap before it finished on its own.\n\n" +
"Training now runs until it stops improving, then deploys its best model. Status:\n" +
reasons +
balancedStr +
"\nContinue training?\n\n" +
"Yes = keep training for another " + IntegerToString(m_maxErasPerRun) + " eras\n" +
"No = deploy the best checkpoint so far and stop training";
int res = MessageBox(msg, "Warrior EA - training", MB_YESNO | MB_ICONQUESTION);
return (res == IDYES);
}
//+------------------------------------------------------------------+
//+------------------------------------------------------------------+
//| See the declaration comment - the single deploy-persistence path. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::PersistDeployedModel(void)
{
if(CheckPointer(Net) == POINTER_INVALID)
return;
double currentIndicatorParams[];
m_indicatorTuner.Flatten(currentIndicatorParams);
if(!Net.Save(m_activeFileName + ".nnw", dError, dUndefine, dForecast, dtStudied, m_activeFileCommon, m_eraCount, m_trainingComplete, currentIndicatorParams))
Print(__FUNCTION__ + ": ERROR - Net.Save failed for " + m_activeFileName + ".nnw. The deployed model was NOT persisted to disk.");
//--- Deploy-time gate: does this model's pure-MQL5 forward pass match the backend? If so, an
//--- inference-only backtest can run DLL-free (see ValidateCpuInference / CNet::SetCpuInference).
//--- Persisted into the .stats written next. Chart-only; safe-false everywhere else.
m_mqlInferenceValidated = ValidateCpuInference();
if(!SaveModelStats(m_activeFileName, m_activeFileCommon)) // keep calibration state paired with the just-saved weights
Print(__FUNCTION__ + ": ERROR - SaveModelStats failed for " + m_activeFileName + ". Calibration state not persisted.");
SaveShadowNet(currentIndicatorParams);
}
//+------------------------------------------------------------------+
void CExpertSignalAIBase::FinalizeTrainRun(void)
{
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- A run stopped mid-pass-2.5 or mid-pass-3 never reached that pass's own unfreeze, so lift the
//--- scoring freeze here before anything else touches the net - the deployed model must adapt live
//--- (see the freeze at pass-3 start, and the identical one the calibration walk takes for the same
//--- reason). Harmless no-op when already unfrozen or on a net with no normalization.
feat: mini-batch gradient accumulation (F4), front-end-aware capacity budget (F6), split Wyckoff categoricals (N1) Completes the 2026-08-09 training audit. FORCES A RETRAIN of every Wyckoff-enabled config (N1 re-keys the fingerprint), and BOTH DLLs must be redeployed alongside the .ex5 - they carry new exports. F4 - mini-batch accumulation, TRAIN_BATCH_SIZE=32. Training was pure online SGD (one weight update per bar), which is the mechanical source of the era-to-era whipsaw every downstream guard was built to cope with. The O(n^2) outer product is native - AccumulateWeightGrad / AccumulateWeightGradConv / AccumulateBufferInto in Network.cl, WarriorCPU and WarriorDML - while the optimizer step is host-side MQL5 shared by all tiers (ApplyAccumToBlock), so there is one Adam/SGD implementation instead of four that can drift. - the LSTM needs no outer-product kernel (WeightsGradient already holds the sample's full dW) but could NOT simply be left un-zeroed between samples: CPU_LSTMSeqBackward/DML_LSTMSeqBackward memset it on entry. Hence a separate accumulator plus an elementwise add. - batch-norm gamma/beta accumulate in host arrays, not new BatchOptions slots - BN_OPT_STRIDE is baked into every persisted .nnw. - scoped to pass 2; online learning keeps immediate updates. Every save / checkpoint / scoring boundary flushes, scaling by the real sample count. - degrades to per-sample updates (one log line) on a tier that cannot accumulate, so old devices and DLL-free builds are unaffected. - verified offline: DirectML/batch_accum_check.cpp drives the real exports against an independent reference; at B=1 the accumulator matches the shipped unbatched kernel's own gradient to 1.1e-16. Math only - the in-situ check remains the per-layer dW/W report on a real era. F6 - ComputeFirstLayerWidth budgeted against the RAW input width even where a conv/LSTM front end had already reduced it, so an LSTM's dense stack was charged for 1,280 inputs when it receives 64. Confirmed from the deployed .cfg files: CONV, LSTM and HYBRID were all pinned at the 16-unit floor. Now budgeted against the front-end output and capped at it (never fan out), with the derivation reordered so both stages settle first. N1 - EventCode/EventPhase/StructuralPhase are signed categoricals packing direction and Wyckoff stage into one scalar across a sign discontinuity. Split into direction + [0,1] magnitude, the same convention the base OHLC block uses. Information-preserving; 13 readings now occupy 16 inputs. Compiled clean (0 errors, 0 warnings); both DLLs rebuilt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 11:48:03 -04:00
//--- Same treatment for a run stopped mid-pass-2, which never reached that pass's flush: apply the
//--- partial batch and drop back to per-sample updates, so the net this function is about to
//--- checkpoint, persist and hand to live inference has nothing accumulated behind it.
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
if(CheckPointer(Net) != POINTER_INVALID)
feat: mini-batch gradient accumulation (F4), front-end-aware capacity budget (F6), split Wyckoff categoricals (N1) Completes the 2026-08-09 training audit. FORCES A RETRAIN of every Wyckoff-enabled config (N1 re-keys the fingerprint), and BOTH DLLs must be redeployed alongside the .ex5 - they carry new exports. F4 - mini-batch accumulation, TRAIN_BATCH_SIZE=32. Training was pure online SGD (one weight update per bar), which is the mechanical source of the era-to-era whipsaw every downstream guard was built to cope with. The O(n^2) outer product is native - AccumulateWeightGrad / AccumulateWeightGradConv / AccumulateBufferInto in Network.cl, WarriorCPU and WarriorDML - while the optimizer step is host-side MQL5 shared by all tiers (ApplyAccumToBlock), so there is one Adam/SGD implementation instead of four that can drift. - the LSTM needs no outer-product kernel (WeightsGradient already holds the sample's full dW) but could NOT simply be left un-zeroed between samples: CPU_LSTMSeqBackward/DML_LSTMSeqBackward memset it on entry. Hence a separate accumulator plus an elementwise add. - batch-norm gamma/beta accumulate in host arrays, not new BatchOptions slots - BN_OPT_STRIDE is baked into every persisted .nnw. - scoped to pass 2; online learning keeps immediate updates. Every save / checkpoint / scoring boundary flushes, scaling by the real sample count. - degrades to per-sample updates (one log line) on a tier that cannot accumulate, so old devices and DLL-free builds are unaffected. - verified offline: DirectML/batch_accum_check.cpp drives the real exports against an independent reference; at B=1 the accumulator matches the shipped unbatched kernel's own gradient to 1.1e-16. Math only - the in-situ check remains the per-layer dW/W report on a real era. F6 - ComputeFirstLayerWidth budgeted against the RAW input width even where a conv/LSTM front end had already reduced it, so an LSTM's dense stack was charged for 1,280 inputs when it receives 64. Confirmed from the deployed .cfg files: CONV, LSTM and HYBRID were all pinned at the 16-unit floor. Now budgeted against the front-end output and capped at it (never fan out), with the derivation reordered so both stages settle first. N1 - EventCode/EventPhase/StructuralPhase are signed categoricals packing direction and Wyckoff stage into one scalar across a sign discontinuity. Split into direction + [0,1] magnitude, the same convention the base OHLC block uses. Information-preserving; 13 readings now occupy 16 inputs. Compiled clean (0 errors, 0 warnings); both DLLs rebuilt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 11:48:03 -04:00
{
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
Net.SetBatchNormFrozen(false);
feat: mini-batch gradient accumulation (F4), front-end-aware capacity budget (F6), split Wyckoff categoricals (N1) Completes the 2026-08-09 training audit. FORCES A RETRAIN of every Wyckoff-enabled config (N1 re-keys the fingerprint), and BOTH DLLs must be redeployed alongside the .ex5 - they carry new exports. F4 - mini-batch accumulation, TRAIN_BATCH_SIZE=32. Training was pure online SGD (one weight update per bar), which is the mechanical source of the era-to-era whipsaw every downstream guard was built to cope with. The O(n^2) outer product is native - AccumulateWeightGrad / AccumulateWeightGradConv / AccumulateBufferInto in Network.cl, WarriorCPU and WarriorDML - while the optimizer step is host-side MQL5 shared by all tiers (ApplyAccumToBlock), so there is one Adam/SGD implementation instead of four that can drift. - the LSTM needs no outer-product kernel (WeightsGradient already holds the sample's full dW) but could NOT simply be left un-zeroed between samples: CPU_LSTMSeqBackward/DML_LSTMSeqBackward memset it on entry. Hence a separate accumulator plus an elementwise add. - batch-norm gamma/beta accumulate in host arrays, not new BatchOptions slots - BN_OPT_STRIDE is baked into every persisted .nnw. - scoped to pass 2; online learning keeps immediate updates. Every save / checkpoint / scoring boundary flushes, scaling by the real sample count. - degrades to per-sample updates (one log line) on a tier that cannot accumulate, so old devices and DLL-free builds are unaffected. - verified offline: DirectML/batch_accum_check.cpp drives the real exports against an independent reference; at B=1 the accumulator matches the shipped unbatched kernel's own gradient to 1.1e-16. Math only - the in-situ check remains the per-layer dW/W report on a real era. F6 - ComputeFirstLayerWidth budgeted against the RAW input width even where a conv/LSTM front end had already reduced it, so an LSTM's dense stack was charged for 1,280 inputs when it receives 64. Confirmed from the deployed .cfg files: CONV, LSTM and HYBRID were all pinned at the 16-unit floor. Now budgeted against the front-end output and capped at it (never fan out), with the derivation reordered so both stages settle first. N1 - EventCode/EventPhase/StructuralPhase are signed categoricals packing direction and Wyckoff stage into one scalar across a sign discontinuity. Split into direction + [0,1] magnitude, the same convention the base OHLC block uses. Information-preserving; 13 readings now occupy 16 inputs. Compiled clean (0 errors, 0 warnings); both DLLs rebuilt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 11:48:03 -04:00
Net.FlushBatch();
Net.SetBatchSize(1);
}
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- deploy the most stable/best-scoring era's weights rather than whatever the run happened to
//--- end on (which may reflect drift after the objective was first hit, or an aborted run). Restore
//--- is now the in-MEMORY snapshot (CNet::RestoreWeights) - see CaptureWeights' note for why the old
//--- file-based restore couldn't work on the CPU-DLL backend.
if(m_haveOosCheckpoint)
{
if(Net.RestoreWeights())
{
dOosForecast = m_bestOosForecast;
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//--- Deploy the checkpoint's operating point alongside its weights - the OOS coverage and
//--- precision this run is about to report were measured with this pair together.
m_dirConfThreshold = m_bestDirConfThreshold;
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//--- Same F3 reset as the mid-run restore: the deployed weights are the checkpoint's, so the
//--- optimizer state that continues from here (online continual learning backprops on this
//--- same net - see OnlineLearnStep) must not be the dead run's momentum.
Net.ResetOptimizerState();
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
RefreshLatestSignal();
fix(deinit): a full model write was running ahead of the cheap cleanup "Abnormal termination" is back, and this time it is not the arrows. The timing names the culprit exactly: 16:02:31.547 OnDeinit: shutting down 16:02:36.003 Abnormal termination <- 4.46 s, MetaTrader gave up 16:02:36.226 chart signals - persisted <- cleanup finished 0.2 s LATE OnDeinit called StopTraining() BEFORE the chart cleanup. StopTraining() finalises an in-flight run, and FinalizeTrainRun() restores the best checkpoint and then persists it - a full ~1MB model write per signal. So the expensive step ran ahead of the cheap bounded one, which is precisely the inversion the shutdown ordering exists to prevent. The previous fix put PersistWeightsOnShutdown last and missed that StopTraining smuggles a second save in at the front. Two changes: Cleanup now runs FIRST, then StopTraining, then the weight save. The visible teardown is cheap and bounded, so it always completes even when everything after it is killed. And the deploy-persist inside FinalizeTrainRun is suppressed during shutdown. RestoreWeights() is an in-MEMORY swap, so the best checkpoint is already the live net by that line, and PersistWeightsOnShutdown writes exactly those weights moments later. The old path wrote the same model twice per signal - eight full writes across four charts - for no benefit. A user-pressed Stop still persists immediately, because nothing else would. Compiles 0 errors / 0 warnings. Build tag deinit-order-v2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 16:06:40 -04:00
//--- NOT during shutdown. RestoreWeights() above is an in-MEMORY swap, so the best checkpoint is
//--- already the live net by this line - and OnDeinit's PersistWeightsOnShutdown() is about to
//--- write exactly those weights anyway. Persisting here too means TWO full ~1MB model writes per
//--- signal on the shutdown path, ahead of the chart cleanup, which is what put OnDeinit over
//--- MetaTrader's budget: measured 4.46 s to "Abnormal termination" on 2026-08-01, with the chart
//--- cleanup completing 0.2 s AFTER the kill. Nothing is lost by skipping it; the same bytes reach
//--- the same file one call later.
if(!m_shutdownInProgress)
PersistDeployedModel();
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
}
//--- Clean up any legacy on-disk checkpoint from an older (file-based) build so it can't linger.
int checkpointFlags = m_activeFileCommon ? FILE_COMMON : 0;
if(FileIsExist(m_activeFileName + "_ckpt.tmp", checkpointFlags))
FileDelete(m_activeFileName + "_ckpt.tmp", checkpointFlags);
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- (dtStudied used to be held back while scoring a throwaway candidate - that marker belongs
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- to the DEPLOYED model's "studied up to" state; a candidate eval must leave it untouched. The
//--- checkpoint block above is already inert in eval mode (m_haveOosCheckpoint stays false).
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
if(m_eraCount > 0)
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
dtStudied = m_lastBarTime;
m_trainRunActive = false;
m_eraResumePending = false;
m_haveOosCheckpoint = false;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
m_checkpointEra = -1; // the joint-checkpoint era stamp goes with the snapshot it describes
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//--- Persist the arrows now drawn on the chart so a deploy/stop survives a later re-add/recompile
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- without a retrain (durable even if the terminal never gets a clean OnDeinit).
fix(deinit): O(n^2) arrow prune blew the shutdown budget and littered 3 charts Reported as "the perceptron correctly cleaned its chart on deinit, the other 3 did not, abnormal termination". Measured from the 2026-08-01 log, time from "OnDeinit: shutting down" to MetaTrader force-terminating: PAI 3.75 s -> survived, chart cleaned CONV 4.71 s -> Abnormal termination LSTM 4.28 s -> Abnormal termination HYBRID 4.16 s -> Abnormal termination In all four the last line printed is the inference census, which is the end of StopTraining() - so the overrun is inside ShutdownChartCleanup(), i.e. between saving the arrows and purging them. The cost is the prune loop at the end of SaveChartSignals(): for(int i = 0; i < prunedCount; i++) ObjectDelete(0, SIG_ARROW_PREFIX + TimeToString(pruned[i])); ObjectDelete is O(objects) on a crowded chart, so this is O(n^2). It was harmless while the model called a direction on ~6% of bars. After the triple-barrier relabel the models call on 83-94% of bars, the chart carries many thousands of arrows, and the loop overran MetaTrader's OnDeinit budget - so PurgeChart() never ran and the arrows stayed on screen. The slow tidy-up starved the fast one. The work was pure waste at that moment: ShutdownChartCleanup purges every arrow with a single bulk ObjectsDeleteAll immediately afterwards. Deleting them one at a time first has no effect except to prevent the bulk delete from happening at all. SaveChartSignals takes a pruneChartObjects flag, and the two shutdown call sites pass false: - ShutdownChartCleanup passes `preserveChartArrows`, which is exactly right: prune when the arrows are STAYING (chart and sidecar must agree), skip when they are about to be purged wholesale. - FinalizeTrainRun passes !m_trainingStopRequested. Removing a chart MID-ERA reaches StopTraining -> FinalizeTrainRun, which took the expensive path a second time, even earlier, before anything had been cleared. Same defect one call site up; it only escaped notice because the observed removals happened to land between eras. Normal convergence and the live per-era path are unchanged - they still prune, which is what keeps the chart object count bounded. This also restores the invariant the 2026-07 fix intended ("chart cleanup runs BEFORE the heavy weight save so a stall cannot leave the chart littered"). That fix moved cleanup ahead of the WEIGHT save, but cleanup had since grown its own slow step ahead of its own fast one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:38:36 -04:00
//--- The prune is suppressed when a STOP is in flight, because that means StopTraining() called us and
//--- ShutdownChartCleanup() is about to bulk-purge every arrow anyway. Without this, removing a chart
//--- MID-ERA takes the expensive path twice: once here and once in the cleanup that follows, both
//--- before anything has been cleared. Same defect as the shutdown prune, one call site earlier - see
//--- the prune block in SaveChartSignals() for the measurement.
SaveChartSignals(!m_trainingStopRequested);
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
}
#endif // WARRIOR_AIBASE_TRAINING_MQH
Copy permalink