Warrior_EA/Expert/ExpertSignalAIBase.mqh

3388 lines
216 KiB
MQL5
Raw Permalink Normal View History

feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//+------------------------------------------------------------------+
//| Warrior_EA |
//| AnimateDread |
//| |
//+------------------------------------------------------------------+
#include "ExpertSignalCustom.mqh"
#include "..\AI\Network.mqh"
#include "..\Variables\IndicatorResources.mqh"
#include "..\Variables\IndicatorTuneRanges.mqh"
#include "..\System\StatusLabel.mqh"
#include "..\System\NewsRelevance.mqh"
fix(signals): revive a dead MA model, and demote Sanyaku from state to event Two defects surfaced by research/test_classic.py, both verified fixed by re-running the transcription against 178k bars of EURUSD H1. CSignalMA model 1 could never fire. For any recursive average - and MA_TYPE_EMA is the shipped default - MA(i) = a*Close(i) + (1-a)*MA(i+1), so DiffMA(i) = a * (Close(i) - MA(i+1)) DiffCloseMA(i) = (1-a) * (Close(i) - MA(i+1)) are positive multiples of one quantity and always share a sign. Model 1 asks for a close BELOW a RISING average, which is precisely the combination that identity forbids: 0.000% of bars, either direction, any symbol. The MQL5 standard library this was ported from defaults to MODE_SMA, where the two are merely correlated - the bug arrived with the EMA default, not with the port. Reading the slope one bar back (DiffMAPrev) breaks the tie for every MA type while keeping the model's stated meaning. Now fires on 7.92% of bars. CSignalIchimoku model 11 fired on 27% of bars at weight 100. Sanyaku is three standing STATES conjoined with no transition term, so it held across long stretches - and being last in the if-chain at the top weight, the module's highest-conviction reading was also its most common one, overwriting all eight event models below it on a quarter of all bars. The old comment rejected an event form because "demanding all three flip on the same bar would fire almost never" - true, but that is not the alternative. Kouten is the TURN: the ALIGNMENT transitions, and only one role need change for it to. Testing !Sanyaku(idx+1) fires once per aligned stretch. Now 2.17%, in line with Kumo breakout (2.4%) and the strong TK cross (1.1%). DataReady() extended one bar deeper to cover the lookback. Neither pattern showed edge before or after; this is about the models meaning what they say and the vote not being dominated by a constant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:14:34 -04:00
#include "..\System\CrossAsset.mqh"
#include "..\System\AltData.mqh"
#include "ADIndicatorTuner.mqh"
refactor(dry): one binomial arithmetic for every "is this edge real" test The formula p(1-p)/n was transcribed nine times across six files - the two deploy gates, the two edge floors, the collapse recall floor, the barrier rung ladder, the inference bin SE, the pooled inverse-variance weights and both detectability reports. System\BinomialStats.mqh now holds it once, as free functions with no class dependency, so the god-class declaration does not grow to host pure math. BinomialVar(p, n) p(1-p)/n BinomialSEPct(p, n) 100*sqrt(p(1-p)/n) BinomialCallsForEdge(p, edge, sigmas) the same, solved for n NormalUpperTailQ(z) Q(z), via Math\Stat\Normal.mqh SidakFamilyP(z, N) 1-(1-Q(z))^N Value-preserving by construction: rates go in as probabilities so no call site gained a *100/100 round-trip, and BinomialSEPct is written through BinomialVar so the multiply order is the one it replaced. Every degenerate guard each site carried (p<=0, p>=1, n<=0) now lives in one place and returns the 0 those sites already treated as "no bar to clear". CExpertSignalAIBase::NormalUpperTail is gone; NormalUpperTailQ replaces it. What consolidating SURFACED, and is deliberately NOT changed here: the two Sidak selection gates compute their SE on the RAW call count, while every other SE in the project deflates by EffectiveSampleSize() for triple- barrier label overlap. That makes them the most permissive test in the codebase, by ~sqrt(mean label lifespan). Correcting it tightens a live deploy bar, which is a policy decision, not a refactor - flagged in the code at both sites. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 19:50:24 -04:00
#include "..\System\BinomialStats.mqh"
refactor(stdlib): one quantile definition, from Math\Stat The codebase had THREE conventions for the same statistic. AltData took a true median; the barrier horizon and the derived input window took the upper of the two middle values; the MI terciles and the barrier stop ladder used nearest-rank indexing. All four now go through MathMedian / MathQuantile, which is R's type 7 and the library's one answer. System\AltData.mqh column median -> MathMedian (exact, no change) AIBase\Labels.mqh swing median -> MathMedian leg-range med -> MathMedian stop ladder -> MathQuantile, read in one call AIBase\Topology.mqh window median -> MathMedian AIBase\AutoTune.mqh MI terciles -> MathQuantile + MathMin/MathMax Signals\SignalSessionFilter DST last Sunday-> CDateTime::DaysInMonth() gaps[]/legs[] change from int to double so MathMedian can read them; the values are bar counts either way. VALUES MOVE. Even-sample medians shift by half a bin and the quantile reads interpolate, so the barrier geometry and the derived input window can land on different rungs - re-keying fingerprints and forcing a retrain. Accepted deliberately: stdlib consistency was the ask, and three private conventions for one statistic is what it buys out. Two YAGNI finds fell out of the ladder rewrite. MathQuantile sorts its own copy, so DeriveBarrierGeometry no longer sorts up[]/dn[] in place - which means upUnsorted[], a full array copy kept only to undo that sort, is gone. ArraySort(up) had no consumer needing order at all; it was pure work. The library call also gets a failure guard the hand-rolled indexing never needed but the ladder read does. Verified while here: Math\Stat\Math.mqh's MathAbs/MathMax/MathSqrt/MathPow and friends are ARRAY overloads, not scalar redefinitions, so pulling it into the translation unit shadows no builtin. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 20:16:03 -04:00
#include <Math\Stat\Math.mqh>
//--- Alglib forest / least-squares for AIBase\Baselines.mqh. Also pulls in statistics.mqh, where the
//--- signal-database ranking's significance tests come from - one include serves both.
feat(baselines): Alglib forest + linear on the NN's own matrix Every direction verdict so far was measured through one architecture family, so "flat" has two readings that no topology tuning can separate: the net is the wrong learner, or the matrix carries no directional information. Two learners with completely different inductive biases - Alglib's random decision forest and an ordinary least-squares fit - now train on the SAME feature windows (BuildFeatureWindow, the net's own function, so there is no second feature implementation to drift), the SAME labels, the SAME IS/OOS split with both purges, and are scored through the SAME precision-against-always-one-direction comparison and the same Sidak family-wise arithmetic the deploy gate uses. If both also land at chance, the matrix is the limit. Deliberate choices, each of which could have made the comparison a different question wearing this one's name: - LRBuild, not LRBuildZ: the intercept absorbs the class imbalance, and a baseline handicapped by a forced zero intercept would flatter the net for the wrong reason. - Raw call counts in the SE, matching the live gate's known-permissive test rather than correcting it here - both sides must face the same bar. - No threshold sweep on the linear fit: a threshold fitted on the slice being scored is the calibration leak the purged band exists to avoid. - Uniform stride when a cap bites, not the newest N rows, so a score difference cannot be a regime difference. What was dropped is logged. Ships off (Run_Alglib_Baselines = false): it is a measurement, not a trading feature, nothing trades on the answer and no model is saved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 23:45:03 -04:00
#include <Math\Alglib\dataanalysis.mqh>
//--- FFT cross-correlation for the all-lags profile. Not reached by dataanalysis.mqh.
feat(baselines): combining-weight fit, MLP cross-validation, all-lags correlation Three ALGLIB additions, all measurement-only and all under the existing Run_Alglib_Baselines switch. MinBLEIC COMBINING WEIGHTS. The live ensemble weights each member by its own pooled holdout win rate - a defensible prior, but not a fit, and nothing has ever asked what mixture minimises error on the bars the members disagreed about. Two individually-mediocre members wrong in different places can beat one individually better, and a per-member win rate cannot express that because it never looks at them jointly. Solved on the simplex (w >= 0, sum w = 1), which is exactly what MinBLEIC is for. Non-negative because a negative weight asserts "trade the opposite of this member", a claim ~60 effective observations cannot support. Least squares on the signed outcome rather than precision: precision is a STEP function of the threshold that no gradient method can walk, and optimising a smooth proxy for a step decision is how c3daded put every operating point 14pp underwater - so the result is reported in BOTH currencies, the SSE it minimised and the directional hit rate the mixture would actually have scored against the equal mix. If the second does not improve, the first is noise. This needed data that did not exist: g_ensVoteSum accumulates member contributions and the sum destroys the decomposition, while g_ensVoteVoterMask records only WHETHER a member voted, never what. g_ensVoteMember[] keeps them unsummed. The live arithmetic is untouched. ENS_MAX_MEMBERS is 8 and deliberately larger than MAX_AI_SIGNALS (5): independent caps, over-allocating is free, and matching them would make this array silently short the day the registry grows - a cap that has already dropped a member once without saying so. MLPKFoldCVLBFGS. Every baseline row carries a binomial SE, which is the sampling error of SCORING a fixed model and says nothing about how much the FIT moves. One LBFGS run from one random start can land anywhere, and a baseline that cleared or missed the bar on luck of initialisation reads exactly like one that did it on merit. 3 folds, because each is a full retrain. LBFGS not LM - LM builds a Hessian over ~7,700 weights. CCorr ALL-LAGS PROFILE. Added BESIDE the MI lag profile, not instead: MI catches nonlinear dependence and is the stronger negative, which is why it settled the verdict - but its per-lag permutation null limits it to ~20 lags. FFT correlation gets every lag in one O(n log n) pass, so linear structure parked at lag 300 would surface for free. Different question, not a replacement. Walks CONTIGUOUS bars, unlike everything else in this file, because a lag index is meaningless otherwise; both series are mean-centred first since CorrR1D is a raw sum of products; and the max over columns x lags is judged against a Sidak family of exactly that size, not a bare 2-sigma line. fasttransforms.mqh needed its own include - verified that none of ap/optimization/statistics/solvers/linalg reaches it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 07:44:36 -04:00
#include <Math\Alglib\fasttransforms.mqh>
//--- Soft one-hot targets for the 3-neuron head. The per-neuron SIGMOID forward pass can saturate before
//--- the softmax normalises, and a literal 1.0/0.0 target it only approaches asymptotically grows
//--- weights toward the MAX_WEIGHT clamp. The excursion head trains on hard 1/0 instead.
#define LABEL_SMOOTH_HIGH 0.9
#define LABEL_SMOOTH_LOW 0.05
//--- Control-panel object namespace. CAppDialog names every control from the dialog name, so one prefix
//--- covers the tree. Declared here so it can appear in the chart-prefix sweep list below.
//--- (SIG_ARROW_PREFIX moved to ExpertSignalCustom.mqh when the classic signals started drawing too.)
fix: purge every EA object namespace on init and after deinit teardown Leftover objects survived deinit because the cleanup list had drifted. PurgeChart()'s own comment said it removed "our namespaced signal arrows plus the status-label objects" while the code removed arrows ONLY, and the panel prefix was swept at OnInit and nowhere else - so an ordinary deinit left the status line, and any panel straggler, on the chart. Three scattered call sites and a comment cannot be kept in step. There is now ONE list - WarriorChartPrefixes() - covering arrows, status label and panel, and one sweep, WarriorPurgeChartObjects(), used by every path. Add a prefix there when a new object family appears and every cleanup picks it up. Two call sites added: OnInit, before ANYTHING is drawn (including the status label it would otherwise delete). Chart objects live in the chart PROFILE, not in the EA, so they outlive the process: a deinit force-terminated at MetaTrader's ~4,500 ms budget, a crash, a terminal kill, or an .ex5 replaced while attached all strand objects no later deinit will ever own - and deleting the EA's files does not remove them, which is why they read as corruption. Arrows are included: LoadChartSignals restores them from their sidecar moments later and already opens with its own arrow sweep, so this only removes orphans the sidecar does not account for - the ones SaveChartSignals would otherwise ADOPT, since it rebuilds that sidecar by scanning the chart. OnDeinit, after ExtPanel.Destroy. Destroy walks an unbounded control tree and ClearStatusLabel clears text rather than guaranteeing object removal; either can leave a straggler and nothing looked afterwards. Bounded work - three prefix deletes and one object-list scan - so it respects the ordering rule that keeps the cheap visible cleanup ahead of the heavy save. Arrows excluded: ShutdownChartCleanup already persisted and removed them and re-deleting would race that write. The two are complementary: the deinit sweep closes the ordinary case, the OnInit purge closes the case where MetaTrader never let us finish. Only the second can help after a starved shutdown. Both sweeps rescan by name across EVERY object type and delete what the bulk call missed. ObjectsDeleteAll's return has already been observed disagreeing with a by-name scan of the same chart microseconds apart, and object commands are queued on the chart rather than applied inline, so a returned count is not evidence the objects are gone. Panel create site now uses WARRIOR_PANEL_PREFIX instead of a literal, so the name cannot drift away from the list that cleans it up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:19:26 -04:00
#define WARRIOR_PANEL_PREFIX "WarriorCP"
//--- Base of the PER-INSTANCE custom event id space for the training "study" event. Per-instance because
//--- a shared id made every member run a train chunk for every other member's event - N*N chunks, and a
//--- completely dead control panel - and because id 1 is the Controls library's own ON_DBL_CLICK.
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
#define STUDY_EVENT_ID_BASE 500
//--- An armed study event this old that never arrived is declared lost and re-armed. Generous: a queued
//--- event can legitimately wait tens of seconds behind a sibling's warm-up diagnostics.
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
#define STUDY_EVENT_LOST_MS 60000
//--- Next unassigned study-event id, claimed in the constructor - numbers this chart's members 0..N-1.
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
int g_warriorStudyEventSeq = 0;
//+------------------------------------------------------------------+
feat(ensemble): per-NN inputs replace the preset selector - the meta head becomes the vote's gate User design (2026-08-19): 'remove the enum menu that selects neural networks... individual inputs for every NN just like classic signals... the META NN should be integrated into the voting decision pipeline when enabled... as a bonus meta labelling is applied to enabled NNs.' - AI_CHOICE is GONE (tombstoned per the stale-.set doctrine). Use_MLP/Use_CONV/Use_LSTM/ Use_CONVLSTM are ordinary bools like the classic votes; the ensemble arithmetic adapts to any subset because the consensus divisor is the enabled capable weight. Two or more enabled = ensemble (|ENS1 token + joint gate, exactly the old AI_HYBRID fingerprints, so existing weight files keep loading); one = the old solo preset; none = classic-only. - Use_MetaLabeling un-couples META from the direction NNs (the old selector made them mutually exclusive). S3 ships: CSignalMETA::LiveMetaGate scores each vote-cleared entry (shared window at bar 1 + proposal descriptor: side, net vote, live geometry, spread/ATR; pattern one-hot ZEROED - ranking, not calibrated probability, documented in the body) and vetoes below the cost-adjusted break-even. Entries only; fail-open everywhere, loudly. - COEXISTENCE HAZARDS closed: VoteCapableWeight()=0 and ProspectiveVote()=false for the meta target - solo-only until today, a trained META would otherwise sit in the consensus divisor as a permanent abstainer and shrink every vote by its module weight. - CERTIFIED == TRADED: the ensemble era verdict replays the identical veto through the same g_warriorMetaGate pointer over its OOS fired bars (bar re-resolved from the row's own time; fail-open counted as fires and reported: 'metaGate: N approved, M vetoed, K unscored'). The overlay deliberately does NOT replay it (veto-filter-in-replay class, calendar-cliff precedent) - documented at the sweep site. Solo charts' own gate does not model the veto - the standing solo-gate caveat, documented at the input. - DB continuity: the pattern/journal DB fingerprint's first slot was (int)AIType; DbLegacyAiSlot() maps every legacy-expressible config to its OLD value (new 2-3 member subsets get 100+bitmask, outside the legacy range) so no existing database re-keys. filterID becomes the enabled roster via one EnabledNNSummary(). - HUD: the meta line shows the gate (armed/(trn), last P vs BE, ok/veto tally); the armed/disarmed announcement fires on state change via one latch (MetaGateArmedNow), not only when an entry happens to be proposed. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 13:01:02 -04:00
//| ENSEMBLE CHART-LEVEL SHARED STATE (2+ direction NNs enabled). |
//| Era barrier, combined-vote OOS score and warm-up sharing are |
//| chart-level because they are questions about what gets TRADED. |
//| Solo charts register nothing and none of it runs. |
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
//+------------------------------------------------------------------+
class CExpertSignalAIBase;
CExpertSignalAIBase *g_warriorEnsemble[];
//--- Combined-vote OOS rows for the current era; a stale-era contribution resets the buffer.
//--- TWO masks: Mask = who EVALUATED this bar, VoterMask = who cast a NON-ZERO vote. VoterMask is the
//--- divisor, because Direction() skips abstentions in both the sum and the count.
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
datetime g_ensVoteTime[];
double g_ensVoteSum[];
//--- The same contributions UNSUMMED, one slot per member per bar - the decomposition g_ensVoteSum
//--- destroys. Read by the combining-weight fit in AIBase\Baselines.mqh.
feat(baselines): combining-weight fit, MLP cross-validation, all-lags correlation Three ALGLIB additions, all measurement-only and all under the existing Run_Alglib_Baselines switch. MinBLEIC COMBINING WEIGHTS. The live ensemble weights each member by its own pooled holdout win rate - a defensible prior, but not a fit, and nothing has ever asked what mixture minimises error on the bars the members disagreed about. Two individually-mediocre members wrong in different places can beat one individually better, and a per-member win rate cannot express that because it never looks at them jointly. Solved on the simplex (w >= 0, sum w = 1), which is exactly what MinBLEIC is for. Non-negative because a negative weight asserts "trade the opposite of this member", a claim ~60 effective observations cannot support. Least squares on the signed outcome rather than precision: precision is a STEP function of the threshold that no gradient method can walk, and optimising a smooth proxy for a step decision is how c3daded put every operating point 14pp underwater - so the result is reported in BOTH currencies, the SSE it minimised and the directional hit rate the mixture would actually have scored against the equal mix. If the second does not improve, the first is noise. This needed data that did not exist: g_ensVoteSum accumulates member contributions and the sum destroys the decomposition, while g_ensVoteVoterMask records only WHETHER a member voted, never what. g_ensVoteMember[] keeps them unsummed. The live arithmetic is untouched. ENS_MAX_MEMBERS is 8 and deliberately larger than MAX_AI_SIGNALS (5): independent caps, over-allocating is free, and matching them would make this array silently short the day the registry grows - a cap that has already dropped a member once without saying so. MLPKFoldCVLBFGS. Every baseline row carries a binomial SE, which is the sampling error of SCORING a fixed model and says nothing about how much the FIT moves. One LBFGS run from one random start can land anywhere, and a baseline that cleared or missed the bar on luck of initialisation reads exactly like one that did it on merit. 3 folds, because each is a full retrain. LBFGS not LM - LM builds a Hessian over ~7,700 weights. CCorr ALL-LAGS PROFILE. Added BESIDE the MI lag profile, not instead: MI catches nonlinear dependence and is the stronger negative, which is why it settled the verdict - but its per-lag permutation null limits it to ~20 lags. FFT correlation gets every lag in one O(n log n) pass, so linear structure parked at lag 300 would surface for free. Different question, not a replacement. Walks CONTIGUOUS bars, unlike everything else in this file, because a lag index is meaningless otherwise; both series are mean-centred first since CorrR1D is a raw sum of products; and the max over columns x lags is judged against a Sidak family of exactly that size, not a bare 2-sigma line. fasttransforms.mqh needed its own include - verified that none of ap/optimization/statistics/solvers/linalg reaches it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 07:44:36 -04:00
double g_ensVoteMember[];
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
int g_ensVoteMask[];
fix(gate): the ensemble gate certified a vote the EA never casts g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on the same criterion the live trade does". It did not. Two independent mismatches, both silent: CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100 number straight off the softmax head. Live contributes m_weight x the tier's pattern weight, and BOTH of those are rewritten from the signal DB by UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share an axis and nothing relates them, so the same bar was one number to the gate and a different one to the order path. Same shape as the 2026-08-09 geometry incident: certified on one game, paid on another. DENOMINATOR. The gate divided by the member count, so an abstaining member pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero contribution in BOTH the sum and the count (`if(direction == 0) continue;` before `number++`) - live is a mean over VOTERS. The gate was therefore scoring a strictly more agreement-heavy set of bars than the EA trades. The contribution hook's own comment asserted the opposite ("abstentions dilute the average exactly as they do in the live vote"), while the AI_CHOICE enum 20 lines away correctly documented union semantics. LiveVoteContribution() is now the single definition of "what this member votes", called from the gate; the live path reaches the same arithmetic through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually voted, separately from who evaluated the bar, because those are the divisor and the shared-population test respectively. ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar wrapper - the OOS scan holds the scanned bar's decision in a local, and dPrevSignal is a different bar. NOT changed, deliberately: the gate still does not model live NMS declustering, and the per-member solo gate still scores every directional call rather than threshold-clearing ones. Both are selection-metric changes and this codebase has twice been bitten by switching one blind. Also corrects a stale paragraph in m_pattern_0's declaration block quoting 80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has been since the confidence floor and alternation gate were removed; the block carried both tables at once, and the dead one was quoted back as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
int g_ensVoteVoterMask[];
//--- The weighted mean's DIVISOR, carried per row rather than recomputed at verdict time: a member's
//--- m_weight can be rewritten by UpdateSignalsWeights() between the scan and the verdict, and the
//--- divisor must be the one in force when the numerator was accumulated.
feat(vote): thresholds become confidence percentages, on ONE scale everywhere User request: "the entry/exit thresholds are manual numbers, I would like them to be confidence percentages, so the current 20 would be only 20% confidence in a profitable trade." WHY 20 WAS EVER SENSIBLE. Under UseDatabaseRanking both factors of a filter's contribution are win rates: the pattern weight is that pattern's measured win rate (UpdateSignalsWeights -> ApplyPatternWeight) and m_weight is the filter's average win rate over its patterns, /100. Dividing the sum by the VOTER COUNT therefore produced a mean of PRODUCTS of two win rates - a genuinely 60%-accurate filter firing a 60% pattern scored 0.60 x 60 = 36. The number was never on a probability scale, so its magnitude meant nothing on its own. Dividing by Sum(m_weight) instead makes it a weighted MEAN of win rates, which is a win rate: result = Sum(w_i*p_i)/Sum(w_i). Every voter at 60% now reads 60; MACD's double-divergence pattern (weight 100) voting alone reads 100. m_weight stops being a discount on the probability and becomes how much a filter's opinion COUNTS - which is what a module weight should always have been. Default Min_Vote_Open 20 -> 50: not a tightening, the same bar re-expressed. ONE SCALE, EVERYWHERE - the part that made this bigger than a rescale. Three other places compared against a 0..1 softmax confidence and would each have become a fresh currency mismatch the moment the input changed meaning: * the AI early-exit route (LiveSignedConfidence vs m_ai_exit_threshold) now reads m_lastAiVote - the AI filters' own weighted mean, undiluted by the classic side, which is the only reason that route exists - against the same m_threshold_close the averaged vote uses. m_ai_exit_threshold is retired rather than left dangling. * m_oosDecisionSeries now carries the vote, not the confidence, so the exit SIMULATION stops modelling a close rule the EA does not run. * ExitPolicy() clamped anything > 1.0 to zero. Passing the unscaled input through that would have silently switched vote exits off in the simulation while live went on running them - found before it shipped; the bound now tracks the scale. LiveSignedConfidence() is deliberately untouched and still 0..1: MM sizing, SL/TP scaling and the intelligent trailing want a model confidence, not a win rate. CALIBRATION CAVEAT, stated in the code where the claim is made: this is only a real probability to the extent the pattern weights are. A pattern with fewer than MIN_TRADES_FOR_WIN_RATE journaled trades keeps its DEFAULT weight - a designed prior (25/50/75/100 for the AI tiers), not a measurement. Until the signal DB fills, "60" means "the designed conviction of the patterns that fired". Closing that gap is the next commit. Also corrects VOTE_CLOSE_PRESETS' comment, which documented the two scales this removes. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:52:08 -04:00
double g_ensVoteWeightSum[];
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
bool g_ensVoteWinLong[];
bool g_ensVoteWinShort[];
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
bool g_ensVoteDirLabel[]; // bar carried a Buy/Sell label - the coverage floor's base rate
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
int g_ensVoteRows = 0;
long g_ensVoteEra = -1;
int g_ensVoteDoneMask = 0;
int g_ensVoteCursor[8]; // per-member monotonic row cursor (members scan bars in the same order)
//--- Deliberately LARGER than MAX_AI_SIGNALS (5): independent caps, and over-allocating is free, whereas
//--- matching would make this array silently too small the day the registry grows.
feat(baselines): combining-weight fit, MLP cross-validation, all-lags correlation Three ALGLIB additions, all measurement-only and all under the existing Run_Alglib_Baselines switch. MinBLEIC COMBINING WEIGHTS. The live ensemble weights each member by its own pooled holdout win rate - a defensible prior, but not a fit, and nothing has ever asked what mixture minimises error on the bars the members disagreed about. Two individually-mediocre members wrong in different places can beat one individually better, and a per-member win rate cannot express that because it never looks at them jointly. Solved on the simplex (w >= 0, sum w = 1), which is exactly what MinBLEIC is for. Non-negative because a negative weight asserts "trade the opposite of this member", a claim ~60 effective observations cannot support. Least squares on the signed outcome rather than precision: precision is a STEP function of the threshold that no gradient method can walk, and optimising a smooth proxy for a step decision is how c3daded put every operating point 14pp underwater - so the result is reported in BOTH currencies, the SSE it minimised and the directional hit rate the mixture would actually have scored against the equal mix. If the second does not improve, the first is noise. This needed data that did not exist: g_ensVoteSum accumulates member contributions and the sum destroys the decomposition, while g_ensVoteVoterMask records only WHETHER a member voted, never what. g_ensVoteMember[] keeps them unsummed. The live arithmetic is untouched. ENS_MAX_MEMBERS is 8 and deliberately larger than MAX_AI_SIGNALS (5): independent caps, over-allocating is free, and matching them would make this array silently short the day the registry grows - a cap that has already dropped a member once without saying so. MLPKFoldCVLBFGS. Every baseline row carries a binomial SE, which is the sampling error of SCORING a fixed model and says nothing about how much the FIT moves. One LBFGS run from one random start can land anywhere, and a baseline that cleared or missed the bar on luck of initialisation reads exactly like one that did it on merit. 3 folds, because each is a full retrain. LBFGS not LM - LM builds a Hessian over ~7,700 weights. CCorr ALL-LAGS PROFILE. Added BESIDE the MI lag profile, not instead: MI catches nonlinear dependence and is the stronger negative, which is why it settled the verdict - but its per-lag permutation null limits it to ~20 lags. FFT correlation gets every lag in one O(n log n) pass, so linear structure parked at lag 300 would surface for free. Different question, not a replacement. Walks CONTIGUOUS bars, unlike everything else in this file, because a lag index is meaningless otherwise; both series are mean-centred first since CorrR1D is a raw sum of products; and the max over columns x lags is judged against a Sidak family of exactly that size, not a bare 2-sigma line. fasttransforms.mqh needed its own include - verified that none of ap/optimization/statistics/solvers/linalg reaches it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 07:44:36 -04:00
#define ENS_MAX_MEMBERS 8
//+------------------------------------------------------------------+
//| Population count over the member masks. Bounded by the 8-slot |
//| ensemble, so a plain loop is both clearest and fastest. |
//+------------------------------------------------------------------+
fix(gate): the ensemble gate certified a vote the EA never casts g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on the same criterion the live trade does". It did not. Two independent mismatches, both silent: CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100 number straight off the softmax head. Live contributes m_weight x the tier's pattern weight, and BOTH of those are rewritten from the signal DB by UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share an axis and nothing relates them, so the same bar was one number to the gate and a different one to the order path. Same shape as the 2026-08-09 geometry incident: certified on one game, paid on another. DENOMINATOR. The gate divided by the member count, so an abstaining member pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero contribution in BOTH the sum and the count (`if(direction == 0) continue;` before `number++`) - live is a mean over VOTERS. The gate was therefore scoring a strictly more agreement-heavy set of bars than the EA trades. The contribution hook's own comment asserted the opposite ("abstentions dilute the average exactly as they do in the live vote"), while the AI_CHOICE enum 20 lines away correctly documented union semantics. LiveVoteContribution() is now the single definition of "what this member votes", called from the gate; the live path reaches the same arithmetic through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually voted, separately from who evaluated the bar, because those are the divisor and the shared-population test respectively. ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar wrapper - the OOS scan holds the scanned bar's decision in a local, and dPrevSignal is a different bar. NOT changed, deliberately: the gate still does not model live NMS declustering, and the per-member solo gate still scores every directional call rather than threshold-clearing ones. Both are selection-metric changes and this codebase has twice been bitten by switching one blind. Also corrects a stale paragraph in m_pattern_0's declaration block quoting 80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has been since the confidence floor and alternation gate were removed; the block carried both tables at once, and the dead one was quoted back as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
int EnsembleBitCount(const int mask)
{
int n = 0;
for(int b = 0; b < 8; b++)
if((mask & (1 << b)) != 0)
n++;
return n;
}
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//+------------------------------------------------------------------+
//| ENSEMBLE DEPLOY GATE. In ensemble mode THE UNIT OF EVALUATION IS |
//| THE VOTE: best era, checkpointing, give-up and deploy all move |
//| here, because all four ask what gets TRADED. |
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//+------------------------------------------------------------------+
double g_ensBestScore = -1.0; // best combined-vote selection score (precision x coverage credit)
bool g_ensBestTradeable = false; // did that era clear the vote's own deployability floor
bool g_ensBestTwoSided = false; // did it fire both long and short
double g_ensBestPrecPct = -1.0; // the winning era's vote win rate, for the family-wise test
double g_ensBestChancePct = -1.0; // and its chance reference
int g_ensBestCalls = 0; // and the n that sets the standard error
long g_ensBestEra = -1;
int g_ensCandidateEras = 0; // N for the family-wise correction: eras that COULD have won
int g_ensErasSinceBest = 0; // shared plateau counter
int g_ensPlateauStage = 0; // shared plateau stage
//--- Which best-era the family-wise deploy test has already run against, -1 = none. Without it the
//--- all-members-plateaued shortcut re-ran the gate against an unchanged best every era, incrementing
//--- the candidate count the correction divides by - the run spent its time RAISING ITS OWN BAR.
fix(gate): the plateau shortcut re-ran the deploy test every era, raising its own bar User report: 'eras since best' in the ensemble line is always 0 (era 147, best at era 90, '0 eras ago'). That is a control-flow bug wearing a display symptom. Once every member's in-sample error had plateaued, the shortcut forced the ladder to its DEPLOY stage on EVERY era. The failed-gate branch resets the stage to 0 so the ladder can climb again - so the shortcut raised it, the branch cleared it, forever. Three consequences, only the first of which was visible: - g_ensErasSinceBest was reset every era, pinning the counter at 0. - The stage-1/2 boosted warm restarts were never reached, so the one mechanism that can un-plateau a stuck member never ran. The models sat at a WORSE error than their best (0.2408 -> 0.3015 on PAI) with no escape. - Every repetition ran EnsembleSurvivesSelection against an unchanged best and incremented the candidate-era count the family-wise correction divides by. The run spent its time RAISING ITS OWN SIDAK BAR - the same waste as the 2026-08-18 inert IS-error stop, one layer up, and the reason a gate that needed >47.8% saw its bar climb era after era. Fix: the shortcut fires ONCE PER BEST-ERA (g_ensGateTestedEra, stamped before the outcome branches because it is the re-running that inflates the family, pass or fail). A refused gate now falls back to the normal counter-driven ladder - warm restart, anneal, then a fresh deploy test - which is the escape the shortcut was skipping. Also, per user: the signal marks were too small to see. Span doubled (2.6 bar widths, so the overhang either side of the candle is ~0.8 bars) and both layers thickened - 1px dotted was invisible on a candle chart at any realistic zoom. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 15:40:52 -04:00
long g_ensGateTestedEra = -1;
//--- One-shot latch so the collective IS-error plateau announces once per run, not once per era.
fix(plateau): the IS-error early stop was inert for every ensemble member 15 hours of training, and the stop that exists to END a run announced itself 1,299 consecutive times without ending anything: SP500 ConvLSTM IN-SAMPLE ERROR PLATEAU - not improved in 1297 / 1298 / 1299 eras (best 0.2689, now 0.3269) ... era 1396, 1397, 1398 SP500 LSTM 536 eras SP500 CONV 442 eras SP500 PAI 150 eras XAUUSD HYB 478 eras XAUUSD LSTM 296 eras XAUUSD CONV 366 eras CAUSE: it wrote its decision into m_plateauStage, and EnsembleEraVerdict mirrors the shared ladder onto every member - `mm.m_plateauStage = g_ensPlateauStage` - on EVERY era, purely so each member's status line shows the collective stage. A display mirror was silently overwriting a decision, so the stop re-armed and re-fired the next era, forever. This is the worst possible direction for this particular bug. Every one of those 1,299 eras was scored out of sample and joined the family the deploy gate corrects over (Sidak, g_ensCandidateEras). The stop's entire purpose is to make that family SMALLER; instead the run spent fifteen hours raising its own bar. - m_isErrorPlateaued: a one-way per-member latch, cleared only by a fresh run. Nothing in the ladder may reset it. The stop condition and the two solo deploy conditions read the latch, not the mirrored stage. - The orchestrator combines: EnsembleEraVerdict requires UNANIMITY across participating members (same participation test the era barrier uses, so an excluded or finished member cannot veto). One member still learning can still move the combined vote, and the vote is what the gate certifies. - Fed in as `dueStage = PLATEAU_STAGE_DEPLOY`, NOT written to g_ensPlateauStage. The block that actually ends the run sits under `dueStage > g_ensPlateauStage`, so assigning the stage directly makes that test false and the deploy never happens - the same inert-write shape as the bug being fixed. Caught before committing; raising dueStage carries it through the ladder's own path (warm restarts skipped, family-wise vote test, measurement screen, joint checkpoint) unchanged. - g_ensIsPlateauAnnounced: announce once per run, not once per era. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 13:09:26 -04:00
bool g_ensIsPlateauAnnounced = false;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
bool g_ensDeployApproved = false; // stage 3 reached AND the vote cleared the family-wise gate
long g_ensLastVerdictEra = -1; // guards against scoring one era twice
//--- Lifetime combined-vote win rate over every bar the VOTE fired on, in the SAME shape as a solo
//--- model's m_cumOosCorrect/m_cumOosTotal so the panel reads identically. Session-scoped like the rest
//--- of the g_ens* ladder state.
long g_ensCumOosCorrect = 0;
long g_ensCumOosTotal = 0;
//--- Mirror of Signal_ThresholdOpen, pushed in at registration so the combined-vote scorer fires on the
//--- same criterion the live trade does. UNITS are the 0..100 VOTE scale, not a confidence percentage -
//--- see LiveVoteContribution(). The seed is only read before registration overwrites it.
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
double g_ensembleVoteThreshold = 60.0;
//--- Set by AdvancePatternDatabaseBackfill() on completion; lets OnTimer bypass the hourly DB-ranking
//--- throttle ONCE so weights refresh from the fresh rows immediately.
bool g_forcePatternWeightsRefresh = false;
//--- Set by RankTiersFromOos() at every pass-3 completion; (re)arms the filtered-view overlay sweep.
//--- A flag rather than an era comparison, because what the sweep needs is "a snapshot just got
//--- fresher" - approximating that with era counters is how it used to re-arm against half-built caches.
fix(chart): display now reads era-end SNAPSHOTS - the live cache is wiped mid-era Full-pipeline analysis after "threshold 30, attained often, nothing drawn, still glued to buy". The log falsified the premise before any code did: 21:40:43 swept 4999, 794 voters, drew 491. Strongest 43.0% vs 30.0% 21:42:07 swept 4999, 0 voters, drew 0 21:51:30 swept 4999, 0 voters, drew 0 21:56:30 swept 4999, 922 voters, drew 382. Strongest 44.0% vs 30.0% The arrows WERE drawn - 491 of them, then 382 - and then erased. ONE root cause, three symptoms: every display path read m_arrowSignalCache, which is wiped to sentinel at each era start and only complete again when pass 3 finishes. With eras at ~30s and a sweep at ~17s: * ARROW FLICKER: a sweep landing mid-era found no voters anywhere, and its else-branch deleted the arrow on every voteless bar - erasing the previous sweep's entire output. The chart cycled populated -> blank -> populated; the user kept catching the blank phase. * READOUT GLUE: the newest-cache walk found only sentinel for ~90% of every era and fell through to dPrevSignal - the frozen purge-band edge bar that reads Buy. 659638e fixed which bar was frozen, not the freezing. * VOTER FLAP: 1299 -> 257 -> 1113 across back-to-back sweeps - each saw a different fraction of half-rebuilt caches. THE FIX, structural rather than another patch: 1. Era-end snapshots. RankTiersFromOos() runs at pass-3 completion - the one moment the cache is complete - and now copies it (raw signals, newest LOOKBACK+16 bars) into member-owned snapshot state, unconditionally, BEFORE its early return: an all-Neutral era is a snapshot worth showing, not an absence of one. Raw signals rather than votes, so a tier re-rank between eras reprices them at read time via LiveVoteContribution for free. 2. The sweep (SnapshotVoteAt) and the prospective readout both read snapshots; the readout's fallback chain is live-cache -> snapshot -> dPrevSignal, and the snapshot leg is the one that fires most of the time. 3. NO DATA IS NOT A VERDICT: a den==0 bar no longer deletes - only an actual sub-threshold vote takes an arrow down. This alone ends the wipe half of the flicker even where snapshots are missing (before the first era). 4. Arming moved from an era-counter diff (which fires at era BOUNDARIES, i.e. precisely when caches are about to be wiped) to g_warriorOverlayArmRequest, set by each RankTiersFromOos - "a member's snapshot just got fresher", the only event a redraw can act on. 60s rate limit collapses the four members' burst into one sweep. Classic-only charts arm once at start. 5. Census now reports the direction split - "922 had a voter (610 buy / 312 sell)" - so "the vote leans buy" is checkable from the log instead of inferred from arrow colours. Also visible in the log and worth knowing: the threshold flip-flopped 30 -> 40 -> 30 across the evening's re-inits (census lines at 21:42-21:51 ran at 40), so part of the observed blankness was configuration, not code. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 22:03:20 -04:00
bool g_warriorOverlayArmRequest = false;
fix(vote): unranked members voted with the stock 25/50/75/100 ladder Two defects behind "arrows drawn while members are still mid-era". 1. THE DRAW. The filtered overlay armed on the FIRST member to finish pass 3 and leaned on a 60 s rate limit to "collapse the burst", assuming members finish seconds apart. They do not - on USDJPY one member was at sample 10496 of pass 2 while another was at 2304, minutes apart. A member with no era-end snapshot returns false from SnapshotVoteAt, and the sweep's `if(!hasData) continue;` skips it BEFORE `den += ModuleWeight()`, so the one finished model's tier weight became the entire vote and was drawn as a consensus arrow. An abstention is a member that looked at the bar and said nothing; a missing snapshot is a member that has not looked. The first must dilute the vote, the second must suppress the draw. The arm is now a readiness MASK - one bit per m_ensembleIndex, set at that member's pass-3 completion, cleared when a sweep arms - and a sweep waits for every enrolled member. Bounded at 10 minutes so a member that stops cannot freeze the chart, and the partial draw PRINTS which members were missing: the be39674 lesson is that a hold must never silence the thing that reports it. 2. THE VOTE ITSELF, which is the worse half and is not display-only. Tier weights are not persisted in the .nnw - they exist only as the output of a completed pass 3 - so before a member's first RankTiersFromOos() it holds the constructor's stock 25/50/75/100. Since 4858507 the vote currency is a WIN RATE, so an unranked tier-3 call enters the capability-weighted mean claiming a 100% win rate beside ranked members contributing ~25. Not a strong opinion: the wrong unit. One unranked member drags the ensemble over any threshold, on every fresh deploy and every resume. USDJPY has a measured ceiling of ~19 and was firing anyway. LiveVoteContribution() now abstains until self-ranked, which drops the member from the sum AND the divisor. One function, so live and the gate move together (2c443ba). Era 0 will therefore report 0 coverage until each member completes one era. The ensemble line says so explicitly rather than leaving it to look like the USDJPY unreachable-threshold case - the two are identical in the coverage number and completely different problems. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-23 08:55:04 -04:00
//--- OVERLAY READINESS, one bit per ensemble member (the bit index IS m_ensembleIndex, which is that
//--- member's slot in g_warriorEnsemble). Set at that member's pass-3 completion, cleared when a
//--- sweep arms.
//---
//--- WHY A MASK AND NOT A RATE LIMIT (2026-08-23): a member with no era-end snapshot returns false
//--- from SnapshotVoteAt, and the sweep's `if(!hasData) continue;` skips it BEFORE the divisor - so
//--- one finished model's tier weight becomes the WHOLE vote and gets drawn as a consensus arrow.
//--- The old code armed on the first member to finish and leaned on a 60 s limit to "collapse the
//--- burst", on the assumption that members finish seconds apart. They do not: on USDJPY one member
//--- was at sample 10496 while another was at 2304 of the same pass, minutes apart, so the sweep ran
//--- with one voter and three silent members and put arrows on the chart for a consensus that did
//--- not exist. AN ABSTENTION IS A MEMBER THAT LOOKED AND SAID NOTHING; A MISSING SNAPSHOT IS A
//--- MEMBER THAT HAS NOT LOOKED. The first must dilute the vote, the second must suppress the draw.
uint g_warriorOverlayReadyMask = 0;
//--- First tick a sweep was wanted but held for a missing member; 0 = not waiting. Bounds the hold,
//--- because a member that stops (converged, stopped, error) would otherwise freeze the chart
//--- forever - the be39674 lesson: a barrier must never silence the thing that reports it.
uint g_warriorOverlayArmSince = 0;
fix: purge every EA object namespace on init and after deinit teardown Leftover objects survived deinit because the cleanup list had drifted. PurgeChart()'s own comment said it removed "our namespaced signal arrows plus the status-label objects" while the code removed arrows ONLY, and the panel prefix was swept at OnInit and nowhere else - so an ordinary deinit left the status line, and any panel straggler, on the chart. Three scattered call sites and a comment cannot be kept in step. There is now ONE list - WarriorChartPrefixes() - covering arrows, status label and panel, and one sweep, WarriorPurgeChartObjects(), used by every path. Add a prefix there when a new object family appears and every cleanup picks it up. Two call sites added: OnInit, before ANYTHING is drawn (including the status label it would otherwise delete). Chart objects live in the chart PROFILE, not in the EA, so they outlive the process: a deinit force-terminated at MetaTrader's ~4,500 ms budget, a crash, a terminal kill, or an .ex5 replaced while attached all strand objects no later deinit will ever own - and deleting the EA's files does not remove them, which is why they read as corruption. Arrows are included: LoadChartSignals restores them from their sidecar moments later and already opens with its own arrow sweep, so this only removes orphans the sidecar does not account for - the ones SaveChartSignals would otherwise ADOPT, since it rebuilds that sidecar by scanning the chart. OnDeinit, after ExtPanel.Destroy. Destroy walks an unbounded control tree and ClearStatusLabel clears text rather than guaranteeing object removal; either can leave a straggler and nothing looked afterwards. Bounded work - three prefix deletes and one object-list scan - so it respects the ordering rule that keeps the cheap visible cleanup ahead of the heavy save. Arrows excluded: ShutdownChartCleanup already persisted and removed them and re-deleting would race that write. The two are complementary: the deinit sweep closes the ordinary case, the OnInit purge closes the case where MetaTrader never let us finish. Only the second can help after a starved shutdown. Both sweeps rescan by name across EVERY object type and delete what the bulk call missed. ObjectsDeleteAll's return has already been observed disagreeing with a by-name scan of the same chart microseconds apart, and object commands are queued on the chart rather than applied inline, so a returned count is not evidence the objects are gone. Panel create site now uses WARRIOR_PANEL_PREFIX instead of a literal, so the name cannot drift away from the list that cleans it up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:19:26 -04:00
//+------------------------------------------------------------------+
//| EVERY chart-object namespace this EA creates, in ONE list - the |
//| scattered call sites drifted and left stragglers behind. Add a |
//| prefix here the moment a new object family appears. |
//| Delete BY PREFIX, never ObjectsDeleteAll: a blanket wipe also |
//| removes the user's own drawings. |
fix: purge every EA object namespace on init and after deinit teardown Leftover objects survived deinit because the cleanup list had drifted. PurgeChart()'s own comment said it removed "our namespaced signal arrows plus the status-label objects" while the code removed arrows ONLY, and the panel prefix was swept at OnInit and nowhere else - so an ordinary deinit left the status line, and any panel straggler, on the chart. Three scattered call sites and a comment cannot be kept in step. There is now ONE list - WarriorChartPrefixes() - covering arrows, status label and panel, and one sweep, WarriorPurgeChartObjects(), used by every path. Add a prefix there when a new object family appears and every cleanup picks it up. Two call sites added: OnInit, before ANYTHING is drawn (including the status label it would otherwise delete). Chart objects live in the chart PROFILE, not in the EA, so they outlive the process: a deinit force-terminated at MetaTrader's ~4,500 ms budget, a crash, a terminal kill, or an .ex5 replaced while attached all strand objects no later deinit will ever own - and deleting the EA's files does not remove them, which is why they read as corruption. Arrows are included: LoadChartSignals restores them from their sidecar moments later and already opens with its own arrow sweep, so this only removes orphans the sidecar does not account for - the ones SaveChartSignals would otherwise ADOPT, since it rebuilds that sidecar by scanning the chart. OnDeinit, after ExtPanel.Destroy. Destroy walks an unbounded control tree and ClearStatusLabel clears text rather than guaranteeing object removal; either can leave a straggler and nothing looked afterwards. Bounded work - three prefix deletes and one object-list scan - so it respects the ordering rule that keeps the cheap visible cleanup ahead of the heavy save. Arrows excluded: ShutdownChartCleanup already persisted and removed them and re-deleting would race that write. The two are complementary: the deinit sweep closes the ordinary case, the OnInit purge closes the case where MetaTrader never let us finish. Only the second can help after a starved shutdown. Both sweeps rescan by name across EVERY object type and delete what the bulk call missed. ObjectsDeleteAll's return has already been observed disagreeing with a by-name scan of the same chart microseconds apart, and object commands are queued on the chart rather than applied inline, so a returned count is not evidence the objects are gone. Panel create site now uses WARRIOR_PANEL_PREFIX instead of a literal, so the name cannot drift away from the list that cleans it up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:19:26 -04:00
//+------------------------------------------------------------------+
int WarriorChartPrefixes(string &out[])
{
feat(chart): on-chart vote readout, and Min_Vote_Open 50 -> 40 THRESHOLD. 40 is a measured correction, not a preference. Once RankTiersFromOos() replaced the designed tier priors with each model's real held-out win rate, the vote converges on that win rate - logged 2026-08-18 as pooled 23-36% across four members on three symbols - so a 50% bar could not be reached by anything on offer and the ensemble gate fired on 0 of 4,865 OOS bars. 40 clears the ~34% break-even those same lines report without being unreachable. The comment says plainly not to copy the number: break-even is a function of the barrier geometry, so read the gate's own "needs >N%" for the config in front of you. READOUT. One line, top-right: VOTE SELL 37.2% peak 44.1% need 40% 3 voter(s) -> no trade Every other number on the chart is downstream of the weighted mean the open threshold is compared against, and that was the one quantity never displayed. A chart with no arrows could mean the models abstained, the vote was diluted, or the threshold is unreachable - and telling those apart meant waiting for an era to end and reading the gate line, which is how the last two sessions went. PEAK is the part that earns its space. A threshold above what the vote ever attains can never fire, and that is not knowable from a single bar - it is precisely the "unreachable gate vs merely unmet gate" confusion this project has paid for twice. Colour carries the verdict rather than the direction: green/red ONLY when the vote would actually place an order, grey otherwise. Green-for-buy would make a below-threshold buy look like a trade, which is the specific misreading the display exists to prevent. Guarded on `total > 0` for the same reason the normalization is: Direction() is inherited as-is by every leaf filter, so without it each filter would write its own opinion into the one shared label and the last to run would win - the reader would be looking at an arbitrary member's number believing it was the vote. Drawn after the +-100 range check, so it shows what the threshold is actually tested against. CORNER_RIGHT_UPPER: the status lines, control panel and ensemble panel all live on the left. Registered in WarriorChartPrefixes() explicitly even though the "Warrior" catch-all already reaches it - that catch-all exists because the list has drifted twice, not to make entries optional. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 18:22:45 -04:00
ArrayResize(out, 7);
fix(ensemble): per-member arrow namespaces; ConvLSTM rename; dialog in purge list The ensemble chart UI had a shared-namespace defect that answered the user question "what do the arrows represent?" with "a bug": all four members drew arrows under the same WarSig_<bartime> object names, so the chart showed whichever member rendered LAST, one member Neutral deleted another member Buy at the same bar, each member init sweep wiped the arrows the previous member had just restored, and SaveChartSignals - which rebuilds the sidecar by SCANNING the chart - persisted every other member arrows into its own history (the exact cross-model laundering its own header warns about, now happening BETWEEN ensemble members). Arrows are now namespaced per member (WarSig_PAI_, WarSig_CONV_, WarSig_LSTM_, WarSig_HYB_): draw, delete, restore, prune, member init sweep, destructor purge and the sidecar scan are all member-scoped, and the tooltip names the model. Global purges keep matching the bare WarSig_ prefix, which covers all member namespaces plus old-format leftovers from earlier builds. Labels: the ensemble panel header no longer says "HYBRID ensemble" (HYBRID is one member; the header is the ensemble) and the CONVLSTM member displays as ConvLSTM instead of Hybrid. Its SHORT id stays HYB deliberately - it names the model folder and changing it would orphan every model trained under that path. Deinit: the alt-data mapping dialog namespace (WarriorAltMap_) joins WarriorChartPrefixes, so both the OnInit purge and the deinit final sweep now cover it - it was in neither list, so a dialog starved of its own Destroy() left its controls on the chart permanently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 18:26:55 -04:00
out[0] = SIG_ARROW_PREFIX; // directional signal arrows - bare prefix, so it also
// matches every per-member namespace (WarSig_PAI_ ...)
fix: purge every EA object namespace on init and after deinit teardown Leftover objects survived deinit because the cleanup list had drifted. PurgeChart()'s own comment said it removed "our namespaced signal arrows plus the status-label objects" while the code removed arrows ONLY, and the panel prefix was swept at OnInit and nowhere else - so an ordinary deinit left the status line, and any panel straggler, on the chart. Three scattered call sites and a comment cannot be kept in step. There is now ONE list - WarriorChartPrefixes() - covering arrows, status label and panel, and one sweep, WarriorPurgeChartObjects(), used by every path. Add a prefix there when a new object family appears and every cleanup picks it up. Two call sites added: OnInit, before ANYTHING is drawn (including the status label it would otherwise delete). Chart objects live in the chart PROFILE, not in the EA, so they outlive the process: a deinit force-terminated at MetaTrader's ~4,500 ms budget, a crash, a terminal kill, or an .ex5 replaced while attached all strand objects no later deinit will ever own - and deleting the EA's files does not remove them, which is why they read as corruption. Arrows are included: LoadChartSignals restores them from their sidecar moments later and already opens with its own arrow sweep, so this only removes orphans the sidecar does not account for - the ones SaveChartSignals would otherwise ADOPT, since it rebuilds that sidecar by scanning the chart. OnDeinit, after ExtPanel.Destroy. Destroy walks an unbounded control tree and ClearStatusLabel clears text rather than guaranteeing object removal; either can leave a straggler and nothing looked afterwards. Bounded work - three prefix deletes and one object-list scan - so it respects the ordering rule that keeps the cheap visible cleanup ahead of the heavy save. Arrows excluded: ShutdownChartCleanup already persisted and removed them and re-deleting would race that write. The two are complementary: the deinit sweep closes the ordinary case, the OnInit purge closes the case where MetaTrader never let us finish. Only the second can help after a starved shutdown. Both sweeps rescan by name across EVERY object type and delete what the bulk call missed. ObjectsDeleteAll's return has already been observed disagreeing with a by-name scan of the same chart microseconds apart, and object commands are queued on the chart rather than applied inline, so a returned count is not evidence the objects are gone. Panel create site now uses WARRIOR_PANEL_PREFIX instead of a literal, so the name cannot drift away from the list that cleans it up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:19:26 -04:00
out[1] = STATUS_LABEL_PREFIX; // status line background + text (System\StatusLabel.mqh)
out[2] = WARRIOR_PANEL_PREFIX; // control panel and its whole control tree
fix(ensemble): per-member arrow namespaces; ConvLSTM rename; dialog in purge list The ensemble chart UI had a shared-namespace defect that answered the user question "what do the arrows represent?" with "a bug": all four members drew arrows under the same WarSig_<bartime> object names, so the chart showed whichever member rendered LAST, one member Neutral deleted another member Buy at the same bar, each member init sweep wiped the arrows the previous member had just restored, and SaveChartSignals - which rebuilds the sidecar by SCANNING the chart - persisted every other member arrows into its own history (the exact cross-model laundering its own header warns about, now happening BETWEEN ensemble members). Arrows are now namespaced per member (WarSig_PAI_, WarSig_CONV_, WarSig_LSTM_, WarSig_HYB_): draw, delete, restore, prune, member init sweep, destructor purge and the sidecar scan are all member-scoped, and the tooltip names the model. Global purges keep matching the bare WarSig_ prefix, which covers all member namespaces plus old-format leftovers from earlier builds. Labels: the ensemble panel header no longer says "HYBRID ensemble" (HYBRID is one member; the header is the ensemble) and the CONVLSTM member displays as ConvLSTM instead of Hybrid. Its SHORT id stays HYB deliberately - it names the model folder and changing it would orphan every model trained under that path. Deinit: the alt-data mapping dialog namespace (WarriorAltMap_) joins WarriorChartPrefixes, so both the OnInit purge and the deinit final sweep now cover it - it was in neither list, so a dialog starved of its own Destroy() left its controls on the chart permanently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 18:26:55 -04:00
out[3] = "WarriorAltMap_"; // alt-data symbol-mapping dialog (ADM_PREFIX in
// Panel\AltDataMapDialog.mqh - literal here because that
// header is included later in the build order)
//--- CATCH-ALL. "Nothing matching our prefixes" and "the chart is clean" are different statements,
//--- and only the first was checked - charts came up with duplicated panels after a purge reported
//--- zero leftovers. Does NOT defeat skipArrows: "WarSig_" does not start with "Warrior".
fix(chart): a purge that reports "zero leftovers" was only ever checking its own list 2026-08-17 21:58: all three charts hit "Abnormal termination" ~5.3 s into OnDeinit with NO cleanup-timings line - the teardown was starved again. The 22:00 init purge then removed 993 / 1373 / 1557 stranded objects and reported ZERO by-name leftovers on every chart, and the charts still came up with duplicated panels. "Nothing matching our prefixes remains" and "the chart is clean" are different statements and only the first was being made. Three changes, in the order they matter: 1. WHY the teardown starved, and it is a gap in ad80e0b. StartLabelCachePrebuild runs ResizeBuffers + RefreshData over the FULL study window (33,984 bars on XAUUSD), unchunked, and OnDeinit cannot begin until it returns. Normally a once-per-run cost. That night SP500 and XAUUSD LSTM were wedged in the "cache invalidated at era start" loop, which calls it on EVERY Train() call - two members re-preparing tens of thousands of bars indefinitely. The terminal closed into that. Guarded now, plus a resumable guard in the prebuild chunk loop (the tally pass after it is not chunked). 2. Catch-all "Warrior" prefix in WarriorChartPrefixes. Every family this EA creates is named Warrior* except the arrows (WarSig_), so one bare prefix covers the three named entries AND anything a rename or a stale .ex5 left under a name nobody remembers. Still a prefix delete, never ObjectsDeleteAll(chart) - the user's own drawings are not ours to remove. Does not defeat skipArrows: "WarSig_" does not start with "Warrior". 3. The init purge now REPORTS the residue it did not claim, by name (up to 12). Not deleted - an unmatched object may belong to the user or another indicator. If a Warrior panel is visible and appears in neither the removed count nor this list, the prefix list has drifted a third time and the name is in the journal instead of being inferred from a screenshot. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 22:07:09 -04:00
out[4] = "Warrior";
//--- Vote arrows, listed SEPARATELY from SIG_ARROW_PREFIX even though the name matches it:
//--- skipArrows protects the per-model arrows because their sidecar is rebuilt by SCANNING them
//--- off the chart.
fix(deinit): vote arrows survived the cheap sweep, and 5 long loops ignored the stop Leftover chart objects on long-history charts. Two causes, one of them introduced by 07aa017. THE ONE I ADDED. The filtered view's overlay draws up to SIGNAL_RESCAN_LOOKBACK_BARS vote arrows. OnDeinit's EARLY VISIBLE-UI SWEEP runs with skipArrows=true, which skips any prefix equal to SIG_ARROW_PREFIX - and "WarSig_VOTE_..." starts with "WarSig_", so every one of them was skipped by the one sweep that is cheap enough to always complete. They then sat in the object list while the two expensive scans that follow walked it: a per-member SaveChartSignals O(total) scan, then the by-name rescan. On a chart with years of history that is thousands of extra objects walked twice, inside a teardown budget measured from the stop REQUEST rather than from OnDeinit's first line. skipArrows exists because the per-model arrows' sidecar is rebuilt by SCANNING them off the chart, so they cannot be deleted before that write. Vote arrows have no sidecar - they are a reconstruction, rebuilt on the next attach - so nothing is preserving them and they now get their own prefix slot, deleted by one native call in the first few milliseconds. THE FIVE LOOPS. A time budget bounds THROUGHPUT, not latency to an unload, and OnDeinit cannot begin until whatever is in flight returns. These all scaled with history and none of them checked: * Training passes 2, 2.5 and 3 yielded only on TRAIN_TIME_BUDGET_MS. Pass 1 has checked IsStopped() all along; the other three never have, and they are the ones that grow with the bar count. Free to fix - the resume state is written either way, so a stopped chunk simply is not re-entered. * PruneDirectionalClusters: the one UNCHUNKED sweep left, once per era over every bar, with its own header noting that raising the training budget cannot help its cost. Now bails outright. * AdvanceChartSignalRestore / AdvanceChartSignalRescan: chunked, but the rescan runs a full feedForward per bar over up to 5000 bars and the restore can hold MAX_RESTORED_ARROWS entries. Checked on the same 64-object stride as the clock read, since the check is not free either. * AdvanceFilteredOverlay (mine, 07aa017) replays Direction() on every classic filter per bar and had no check at all. Now per bar. ChartUI.mqh had ZERO shutdown checks across six loops before this. Nothing was added to the purge path itself: that is the work that must complete, and an IsStopped() check inside it would abort unconditionally - IsStopped() is already true by the time OnDeinit runs. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 17:13:50 -04:00
out[5] = SIG_VOTE_PREFIX;
//--- The vote readout. Already covered by the catch-all, and listed anyway: the catch-all exists
//--- because this list has drifted twice, not to make entries optional.
feat(chart): on-chart vote readout, and Min_Vote_Open 50 -> 40 THRESHOLD. 40 is a measured correction, not a preference. Once RankTiersFromOos() replaced the designed tier priors with each model's real held-out win rate, the vote converges on that win rate - logged 2026-08-18 as pooled 23-36% across four members on three symbols - so a 50% bar could not be reached by anything on offer and the ensemble gate fired on 0 of 4,865 OOS bars. 40 clears the ~34% break-even those same lines report without being unreachable. The comment says plainly not to copy the number: break-even is a function of the barrier geometry, so read the gate's own "needs >N%" for the config in front of you. READOUT. One line, top-right: VOTE SELL 37.2% peak 44.1% need 40% 3 voter(s) -> no trade Every other number on the chart is downstream of the weighted mean the open threshold is compared against, and that was the one quantity never displayed. A chart with no arrows could mean the models abstained, the vote was diluted, or the threshold is unreachable - and telling those apart meant waiting for an era to end and reading the gate line, which is how the last two sessions went. PEAK is the part that earns its space. A threshold above what the vote ever attains can never fire, and that is not knowable from a single bar - it is precisely the "unreachable gate vs merely unmet gate" confusion this project has paid for twice. Colour carries the verdict rather than the direction: green/red ONLY when the vote would actually place an order, grey otherwise. Green-for-buy would make a below-threshold buy look like a trade, which is the specific misreading the display exists to prevent. Guarded on `total > 0` for the same reason the normalization is: Direction() is inherited as-is by every leaf filter, so without it each filter would write its own opinion into the one shared label and the last to run would win - the reader would be looking at an arbitrary member's number believing it was the vote. Drawn after the +-100 range check, so it shows what the threshold is actually tested against. CORNER_RIGHT_UPPER: the status lines, control panel and ensemble panel all live on the left. Registered in WarriorChartPrefixes() explicitly even though the "Warrior" catch-all already reaches it - that catch-all exists because the list has drifted twice, not to make entries optional. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 18:22:45 -04:00
out[6] = VOTE_HUD_PREFIX;
return 7;
fix: purge every EA object namespace on init and after deinit teardown Leftover objects survived deinit because the cleanup list had drifted. PurgeChart()'s own comment said it removed "our namespaced signal arrows plus the status-label objects" while the code removed arrows ONLY, and the panel prefix was swept at OnInit and nowhere else - so an ordinary deinit left the status line, and any panel straggler, on the chart. Three scattered call sites and a comment cannot be kept in step. There is now ONE list - WarriorChartPrefixes() - covering arrows, status label and panel, and one sweep, WarriorPurgeChartObjects(), used by every path. Add a prefix there when a new object family appears and every cleanup picks it up. Two call sites added: OnInit, before ANYTHING is drawn (including the status label it would otherwise delete). Chart objects live in the chart PROFILE, not in the EA, so they outlive the process: a deinit force-terminated at MetaTrader's ~4,500 ms budget, a crash, a terminal kill, or an .ex5 replaced while attached all strand objects no later deinit will ever own - and deleting the EA's files does not remove them, which is why they read as corruption. Arrows are included: LoadChartSignals restores them from their sidecar moments later and already opens with its own arrow sweep, so this only removes orphans the sidecar does not account for - the ones SaveChartSignals would otherwise ADOPT, since it rebuilds that sidecar by scanning the chart. OnDeinit, after ExtPanel.Destroy. Destroy walks an unbounded control tree and ClearStatusLabel clears text rather than guaranteeing object removal; either can leave a straggler and nothing looked afterwards. Bounded work - three prefix deletes and one object-list scan - so it respects the ordering rule that keeps the cheap visible cleanup ahead of the heavy save. Arrows excluded: ShutdownChartCleanup already persisted and removed them and re-deleting would race that write. The two are complementary: the deinit sweep closes the ordinary case, the OnInit purge closes the case where MetaTrader never let us finish. Only the second can help after a starved shutdown. Both sweeps rescan by name across EVERY object type and delete what the bulk call missed. ObjectsDeleteAll's return has already been observed disagreeing with a by-name scan of the same chart microseconds apart, and object commands are queued on the chart rather than applied inline, so a returned count is not evidence the objects are gone. Panel create site now uses WARRIOR_PANEL_PREFIX instead of a literal, so the name cannot drift away from the list that cleans it up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:19:26 -04:00
}
//+------------------------------------------------------------------+
//| Delete every object in those namespaces from a chart, and verify. |
//| skipArrows spares the arrows for the one caller that must: a |
//| re-init restores them from their sidecar, so wiping them flickers.|
//| The rescan is required - object commands are QUEUED, so a bulk |
//| call's return value is not evidence they are gone. Names are |
//| collected before deleting: deleting while enumerating renumbers |
//| the list being walked. |
fix: purge every EA object namespace on init and after deinit teardown Leftover objects survived deinit because the cleanup list had drifted. PurgeChart()'s own comment said it removed "our namespaced signal arrows plus the status-label objects" while the code removed arrows ONLY, and the panel prefix was swept at OnInit and nowhere else - so an ordinary deinit left the status line, and any panel straggler, on the chart. Three scattered call sites and a comment cannot be kept in step. There is now ONE list - WarriorChartPrefixes() - covering arrows, status label and panel, and one sweep, WarriorPurgeChartObjects(), used by every path. Add a prefix there when a new object family appears and every cleanup picks it up. Two call sites added: OnInit, before ANYTHING is drawn (including the status label it would otherwise delete). Chart objects live in the chart PROFILE, not in the EA, so they outlive the process: a deinit force-terminated at MetaTrader's ~4,500 ms budget, a crash, a terminal kill, or an .ex5 replaced while attached all strand objects no later deinit will ever own - and deleting the EA's files does not remove them, which is why they read as corruption. Arrows are included: LoadChartSignals restores them from their sidecar moments later and already opens with its own arrow sweep, so this only removes orphans the sidecar does not account for - the ones SaveChartSignals would otherwise ADOPT, since it rebuilds that sidecar by scanning the chart. OnDeinit, after ExtPanel.Destroy. Destroy walks an unbounded control tree and ClearStatusLabel clears text rather than guaranteeing object removal; either can leave a straggler and nothing looked afterwards. Bounded work - three prefix deletes and one object-list scan - so it respects the ordering rule that keeps the cheap visible cleanup ahead of the heavy save. Arrows excluded: ShutdownChartCleanup already persisted and removed them and re-deleting would race that write. The two are complementary: the deinit sweep closes the ordinary case, the OnInit purge closes the case where MetaTrader never let us finish. Only the second can help after a starved shutdown. Both sweeps rescan by name across EVERY object type and delete what the bulk call missed. ObjectsDeleteAll's return has already been observed disagreeing with a by-name scan of the same chart microseconds apart, and object commands are queued on the chart rather than applied inline, so a returned count is not evidence the objects are gone. Panel create site now uses WARRIOR_PANEL_PREFIX instead of a literal, so the name cannot drift away from the list that cleans it up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:19:26 -04:00
//+------------------------------------------------------------------+
int WarriorPurgeChartObjects(long chartID, bool skipArrows, int &leftoverCount)
{
string prefixes[];
int n = WarriorChartPrefixes(prefixes);
int removed = 0;
leftoverCount = 0;
for(int p = 0; p < n; p++)
{
if(skipArrows && prefixes[p] == SIG_ARROW_PREFIX)
continue;
int r = ObjectsDeleteAll(chartID, prefixes[p]);
if(r > 0)
removed += r;
}
//--- Typed-blind rescan across EVERY object type: filtering on OBJ_ARROW made this blind in the same
//--- way the bulk delete was, which is how two scans of one chart disagreed for three sessions.
fix: purge every EA object namespace on init and after deinit teardown Leftover objects survived deinit because the cleanup list had drifted. PurgeChart()'s own comment said it removed "our namespaced signal arrows plus the status-label objects" while the code removed arrows ONLY, and the panel prefix was swept at OnInit and nowhere else - so an ordinary deinit left the status line, and any panel straggler, on the chart. Three scattered call sites and a comment cannot be kept in step. There is now ONE list - WarriorChartPrefixes() - covering arrows, status label and panel, and one sweep, WarriorPurgeChartObjects(), used by every path. Add a prefix there when a new object family appears and every cleanup picks it up. Two call sites added: OnInit, before ANYTHING is drawn (including the status label it would otherwise delete). Chart objects live in the chart PROFILE, not in the EA, so they outlive the process: a deinit force-terminated at MetaTrader's ~4,500 ms budget, a crash, a terminal kill, or an .ex5 replaced while attached all strand objects no later deinit will ever own - and deleting the EA's files does not remove them, which is why they read as corruption. Arrows are included: LoadChartSignals restores them from their sidecar moments later and already opens with its own arrow sweep, so this only removes orphans the sidecar does not account for - the ones SaveChartSignals would otherwise ADOPT, since it rebuilds that sidecar by scanning the chart. OnDeinit, after ExtPanel.Destroy. Destroy walks an unbounded control tree and ClearStatusLabel clears text rather than guaranteeing object removal; either can leave a straggler and nothing looked afterwards. Bounded work - three prefix deletes and one object-list scan - so it respects the ordering rule that keeps the cheap visible cleanup ahead of the heavy save. Arrows excluded: ShutdownChartCleanup already persisted and removed them and re-deleting would race that write. The two are complementary: the deinit sweep closes the ordinary case, the OnInit purge closes the case where MetaTrader never let us finish. Only the second can help after a starved shutdown. Both sweeps rescan by name across EVERY object type and delete what the bulk call missed. ObjectsDeleteAll's return has already been observed disagreeing with a by-name scan of the same chart microseconds apart, and object commands are queued on the chart rather than applied inline, so a returned count is not evidence the objects are gone. Panel create site now uses WARRIOR_PANEL_PREFIX instead of a literal, so the name cannot drift away from the list that cleans it up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:19:26 -04:00
int total = ObjectsTotal(chartID, -1, -1);
string leftovers[];
int found = 0;
if(total > 0)
{
ArrayResize(leftovers, total);
for(int i = 0; i < total; i++)
{
string nm = ObjectName(chartID, i, -1, -1);
for(int p = 0; p < n; p++)
{
if(skipArrows && prefixes[p] == SIG_ARROW_PREFIX)
continue;
if(StringFind(nm, prefixes[p]) == 0)
{
leftovers[found++] = nm;
break;
}
}
}
}
for(int i = 0; i < found; i++)
ObjectDelete(chartID, leftovers[i]);
leftoverCount = found;
return removed + found;
}
//--- Guard against a corrupt .arrows header declaring a garbage count. Restoring is chunked across timer
//--- calls regardless, so a large-but-valid count costs progressive fill-in, never a frozen OnInit.
#define MAX_RESTORED_ARROWS 50000
//--- How many of the MOST RECENT arrows stay on the chart and in the sidecar. Both save and load select
//--- by TIME, not scan order - ObjectsTotal() order is arbitrary, so "the last N scanned" would keep a
//--- random subset rather than the newest.
#define MAX_PERSISTED_ARROWS 1000
//--- Prior strength RankTiersFromOos() shrinks each tier toward the pooled holdout win rate,
//--- counted in EFFECTIVE observations. A tier carries ~8-15 of those per era, so at 10 it sits
//--- about half on its own evidence.
feat(rank): AI models rank their own confidence tiers from held-out outcomes Closes the caveat 4858507 shipped with: the vote is a confidence percentage, but only to the extent the pattern weights are measured. AI tier weights sat at their designed defaults (25/50/75/100) because AI rows only ever arrive from LIVE journaling, of which a training run produces almost none. AND A STALE-TIER BUG THAT MADE THE EVIDENCE MEANINGLESS. The OOS scan bucketed every scanned bar by ConfidenceTier(), which reads dPrevSignal - and dPrevSignal is assigned in PASS 1 only, never anywhere in the OOS scan. So an entire era's fires were bucketed by one stale, unrelated bar's confidence and landed in a SINGLE tier. That is the "tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0)" symptom recorded on 2026-08-16 and attributed to the calibration clamp. The clamp was real and was fixed then; this is a second, independent cause of the identical output that survived that fix untouched - which is why the log kept reading the same afterwards. Two causes, one symptom. Now ConfidenceTierFor(adjSig): the bar this iteration actually scored. WHY THIS DOES NOT WRITE ROWS TO THE SIGNAL DB, which was the obvious reading of "fill the database during training". The user's own observation is the reason: a classic Pattern_2 is a fixed geometric condition, so its win rate is legitimately accumulated over years, but an AI Pattern_2 means "confidence landed in tier 2" and tier 2 under era 100's weights is a different statement from tier 2 under era 500's. The DB's value is ACCUMULATION, and accumulation is exactly what is wrong here - it would average together models that no longer exist, while colliding with the per-table row cap and mixing measured-on-holdout outcomes into the live ledger's own tables. What the DB actually supplies is a measured win rate per pattern, and pass 3 already computes that on held-out bars, thousands at a time. So the model ranks itself once per era, REPLACING rather than accumulating, which makes the weights describe the current weights by construction. ESTIMATOR. Not WinRateFromCounts(): it returns NO_DATA below 100 raw trades BEFORE shrinking, which here would fire on every tier every era and hand all four the pooled rate - the tiers could never separate and the mechanism would be inert. Shrinkage is the answer to a small sample; a floor in front of it means the shrinkage never runs. Instead: a Beta prior of TIER_PRIOR_EFF_N pseudo-observations centred on the model's pooled holdout rate, counted in EFFECTIVE observations, because overlapping triple-barrier labels mean 800 raw fires can be worth ~12 independent ones. Rounded to the integer, not to the decade NormalizeWinRate() uses, which would collapse the shrunk tiers back into one number. NO SAME-ERA CIRCULARITY, and it falls out of the ordering rather than a guard: weights are computed at the END of era N, so the vote scored during era N was cast with era N-1's weights. The deploy gate never grades a vote whose weights were fitted on the bars it is scoring. Residual leakage remains - the same OOS bars each era under a different model - and is stated in the code rather than papered over. Both DB clobber paths are closed: ApplyPatternWeight() declines once self-ranked, and UpdateSignalsWeights()' filter.Weight() call is guarded by SelfRanked() - guarding only the tiers would have let the hourly ranking pass undo half the self-ranking. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 16:00:32 -04:00
#define TIER_PRIOR_EFF_N 10.0
fix(geometry): a free zero made "never resolve" the winning geometry The CANDIDATE GEOMETRY line shipped in 05f1a53 said per-candidate geometry beats the global pair on every SP500 member at 2-3 sigma. It does not. It said so because a bar that reached neither barrier scored 0 R, and the incumbent's mean is NEGATIVE (-0.07 to -0.21 R). Against a losing baseline a free zero is a win, so the widest candidate always came out ahead - and the reported gain ordered itself by timeout share, not by skill: PAI 95.1% timed out -> +0.189 R (head measured -2.42 sigma, HARMFUL) HYB 73.8% -> +0.182 R (head at chance, +0.68 sigma) CONV 61.8% -> +0.163 R (head measured -2.47 sigma, HARMFUL) LSTM 27.1% -> +0.158 R (head +1.67 sigma) Monotone in the timeout share and inverted against the sigma gate. The acceptance test written when this was built - "the sigma gate predicts LSTM helps and CONV hurts; if the R difference does not reproduce that ordering, something is wrong" - is what caught it. A trade that reaches neither barrier is not worth zero. It is closed at the horizon, which is what the scheduled close-all does live and what SimulateTradeOutcome's timeout path already charges. So mark it there: TripleBarrierLabel now publishes the signed close-to-close travel at the last bar it actually visited (m_termTravelCache, same validity flag as the excursion and ladder caches), and LadderOutcomeR prices a timeout off it instead of returning false. A bar that cannot be evaluated under BOTH pairs is now dropped whole - scoring one leg and defaulting the other is the same bug in a smaller costume. Second defect, same function: CandidateGeometryFor applied neither of the floors the global derivation applies, so on USDJPY it chose stop 2.00 / target 1.00 - a 67% break-even, forbidden by the 1:2 policy floor. c3daded in miniature: a selector optimising its own criterion with no reference to the decision criterion. Both floors now apply, and the ratio is re-checked AFTER the per-leg rung snap, which can lose it. Also: the module weight was an unshrunk pooled win rate. USDJPY ConvLSTM fired 19 times (2.0 effective), won 36.8%, and took module weight 0.37 - 41% of the ensemble's capable weight and the loudest voice on the chart, off two effective observations. It also lifted the computed vote ceiling to 26.3 against a 25 threshold, which is why THRESHOLD UNREACHABLE never printed on a chart whose peak vote is 14 and whose practical ceiling without that member is 18.8. The pooled rate is now shrunk toward the coin-flip rate on the era's own OOS bars over 30 prior-equivalent calls, and the tiers shrink toward the shrunk value rather than the raw one. A member with ~300 effective calls moves by ~0.4pp; the 19-fire member goes 0.37 -> ~0.15. MEASUREMENT ONLY still - no order reads any of this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 11:30:54 -04:00
//--- Prior strength for the MODULE weight - how loudly this member speaks in the ensemble mean.
//--- Deliberately far heavier than the tier prior because it is shrunk toward CHANCE, not toward
//--- the member's own pooled rate: a member with almost no held-out fires must not be trusted at
//--- whatever those few fires happened to show. Measured 2026-08-22, USDJPY: ConvLSTM fired 19
//--- times (2.0 effective), won 36.8%, and took module weight 0.37 - 41% of the whole ensemble's
//--- capable weight, off two effective observations, and the loudest voice on the chart. At 30 it
//--- pulls that to ~0.15 while leaving a member with 300 effective calls essentially untouched.
#define MODULE_PRIOR_EFF_N 30.0
//--- Wall-clock budget per chunk of the deferred arrow restore. MQL5 gives a chart ONE thread, so "async"
//--- means small time-boxed slices, never one long blocking pass. 50ms sits between the training chunk
//--- (120ms) and the 500ms timer period.
#define ARROW_RESTORE_BUDGET_MS 50
//--- Max |diff| between the compute backend and the pure-MQL5 path for a model to be marked
//--- MQL5-inference-safe. Summation-order noise is ~1e-6; a genuine port bug shows up as >0.01.
#define CPU_INFERENCE_MAX_DIFF 1.0e-3
//--- Equal-frequency bins the feature column is discretised into. MI is biased upward as bins increase,
//--- and 8 against MI_SAMPLE_BARS keeps that bias small and EQUAL across candidates - equal is what
//--- matters, since this score is only ever used to RANK.
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
#define MI_BINS 8
//--- Bars sampled per candidate. Tuning cost is candidates x this x features, so it is the one number
//--- that trades accuracy for time.
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
#define MI_SAMPLE_BARS 2000
#define MI_MIN_SAMPLES 200
//--- Eras the MI diagnostics may wait for the cross-asset panel before reporting without it.
#define MI_REPORT_MAX_DEFERRALS 3
//--- Largest |k| the label-alignment scan uses. Padding by |offset| instead shifted the offset
//--- build's starting bar and declared every sound measurement void.
#define MI_ALIGN_MAX_SHIFT 5
//--- Coordinate-descent passes; the loop breaks as soon as a pass changes nothing, so this is a ceiling.
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
#define MI_TUNE_PASSES 2
//--- Null draws for the observed score. Empirical p cannot go below 1/(B+1), so 200 reports
//--- "p<=0.005" and no finer. Cheap: BuildMiSample runs ONCE and every draw reuses it. Counting
//--- ranks estimates no spread.
diag(autotune): five permutations was still a coin flip - use a real test The 5-draw z-score shipped an hour ago disproved itself on its first run. All four charts scored the IDENTICAL 0.00401 nats on identical features and identical labels - and reported z of +1.3, +2.0, +4.0 and +4.7. Two "AT THE NOISE FLOOR", two "a real association", same data. The entire swing came from estimating the null's spread from five draws, where the standard deviation of the standard-deviation estimate is ~35%: the denominator was noisier than the effect it was judging. Replaced with an empirical permutation test. 200 draws, p counted by rank with the +1/(B+1) correction (Phipson & Smyth 2010) so p is never reported as exactly zero - no normality assumption and no spread to estimate. The strongest single column is tested against the null distribution OF THE MAXIMUM, which corrects for scoring 26 features at once by construction and is far less conservative than Bonferroni. Affordable because BuildMiSample is now split out of ScoreCurrentParamsByMI and runs ONCE for the whole test - every draw reuses that sample and costs a relabel plus 26 histogram passes, not 2000 feature extractions. The coordinate sweep still calls the combined form, which is correct there: each candidate changes the indicator settings, so its features really do have to be re-extracted. The verdict line keeps both questions apart and prints both answers: the p-value for "is it real", the excess as a percentage of H(Y) for "is it big enough to trade". At n=2000 those can disagree, and collapsing them into one word is how a worthless effect gets called a discovery. Compiles 0 errors / 0 warnings. Build tag permtest-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:45:46 -04:00
#define MI_NOISE_PERMUTATIONS 200
//--- Far below MI_NOISE_PERMUTATIONS because the profile redraws its null at EVERY lag, so cost is
//--- draws x historyBars. 40 resolves p=0.05 to about one draw, and this figure never gates a trade.
diag: MI feature-lag profile - close the blind spot in every MI verdict so far BuildMiSample samples features from ONE bar. So every "MI is at the noise floor" result this codebase has produced - including yesterday's p=0.18 on SP500 H1 - described the ENTRY BAR's 31 features only, while the network is fed 20 bars of them. If information lived at lag 7 and not lag 0, the report would have said "no signal" while the model could still learn. The diagnostic we have been making decisions on had a blind spot exactly the width of the input vector. Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as the existing labelBarOffset and is not interchangeable with it. Shifting the LABEL changes which trade is predicted, so at any non-zero offset the features sit inside the labelled window and the score is lookahead - that is precisely what the alignment scan measures and correctly reports (4.7x more knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label pinned to the entry bar, so every row stays causal. ReportFeatureLagProfile() then scores k = 0..historyBars against the same block-permutation null and reports the deepest lag that clears it - the lookback the data supports, versus the 20 that was picked by hand and never measured. The null is redrawn PER LAG: finite-sample MI bias moves with the realised class counts and bin occupancy, and different rows survive the validity checks at each lag, so one shared floor would be right for lag 0 and wrong everywhere else. Draw count is reduced accordingly (40, not 200) since cost is draws x historyBars; this figure decides a lookback, never a trade. MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that makes two builds comparable row by row. Read-only - no input, topology or label change, so no retrain. Both builds 0/0. Build tag lag-profile-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 13:51:39 -04:00
#define MI_LAG_PERMUTATIONS 40
//--- Per-lag significance, applied against the null of the MAXIMUM over lags rather than each lag's own.
//--- The latter stars one lag per run before any signal exists - on SP500 H1 it produced two opposite
//--- verdicts on identical data hours apart.
diag: MI feature-lag profile - close the blind spot in every MI verdict so far BuildMiSample samples features from ONE bar. So every "MI is at the noise floor" result this codebase has produced - including yesterday's p=0.18 on SP500 H1 - described the ENTRY BAR's 31 features only, while the network is fed 20 bars of them. If information lived at lag 7 and not lag 0, the report would have said "no signal" while the model could still learn. The diagnostic we have been making decisions on had a blind spot exactly the width of the input vector. Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as the existing labelBarOffset and is not interchangeable with it. Shifting the LABEL changes which trade is predicted, so at any non-zero offset the features sit inside the labelled window and the score is lookahead - that is precisely what the alignment scan measures and correctly reports (4.7x more knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label pinned to the entry bar, so every row stays causal. ReportFeatureLagProfile() then scores k = 0..historyBars against the same block-permutation null and reports the deepest lag that clears it - the lookback the data supports, versus the 20 that was picked by hand and never measured. The null is redrawn PER LAG: finite-sample MI bias moves with the realised class counts and bin occupancy, and different rows survive the validity checks at each lag, so one shared floor would be right for lag 0 and wrong everywhere else. Draw count is reduced accordingly (40, not 200) since cost is draws x historyBars; this figure decides a lookback, never a trade. MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that makes two builds comparable row by row. Read-only - no input, topology or label change, so no retrain. Both builds 0/0. Build tag lag-profile-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 13:51:39 -04:00
#define MI_LAG_ALPHA 0.05
//--- WHICH TARGET BuildMiSample() scores against. "Optimal SL/TP" decomposes into HOW FAR price
//--- travels (volatility - predictable) and WHICH barrier is reached first (direction - at the
//--- noise floor).
feat(geometry): 1:2 becomes a FLOOR the swing legs may raise, and the scan can no longer undercut it Two coupled changes, both from measurements in today's SP500 H4 log. 1. THE SCAN WAS OVERRIDING THE DERIVER ON THE WRONG OBJECTIVE. At 13:37:47 DeriveBarrierGeometry produced stop 1.21*ATR / target 2.41*ATR - break-even 33.3%. Thirty-seven seconds later the barrier-geometry scan adopted 2:2 - break-even 50.9% - because it carried 0.0143 nats of entry-time information against the configured pair's 0.0075, and cleared its family-wise gate. Information is not expectancy, and the scan says so itself; nothing checked what the adoption did to the operating point. It did this: the fitted thresholds immediately after read 38.8% win vs 50.9% break-even (-12.2pp) and 48.3% vs 50.9% (-2.6pp), where the earlier model on this instrument at a 1:2 geometry had fitted +1.8pp. The deriver applies the ratio as user RISK POLICY; a scan that can crown 1:1 makes two subsystems disagree about one geometry - the same split this file already fixed once for the clamped-horizon rule. The scan now enrols and crowns only pairings at or above the floor; sub-floor pairs are still scored and printed (marked 'r') so the choice stays auditable. This is NOT the min-RR rule removed on 2026-08-09 - that one guarded a rejection filter that no longer exists. 2. THE RATIO IS A FLOOR, NOT A CAP (user: "the ratio of 1:2 is a minimum that I want, but it should not cap to that if the average zigzag moves gives more room"). BARRIER_TARGET_RR -> BARRIER_TARGET_RR_MIN. ComputeBarrierHorizonBars already scanned ZigZag pivots for leg DURATION; it now harvests leg RANGE in the same pass - two properties of one object, so the horizon and the target describe the same legs instead of two windows. The per-rung ratio is the floor raised toward median-leg/stop, snapped DOWN to a coarse ladder (2.0/2.5/3.0/4.0/5.0). The ladder is coarse on purpose: PooledGate pools only instruments whose structural break-even matches, and continuous per-instrument ratios would never match and would silently empty the pool. A leg is the right yardstick precisely because it owes NOTHING to the barrier - sizing a target off travel measured over the barrier's own horizon is the circular loop that ran EURUSD/USDCAD away to 14-31*ATR in 2026-08-07. The raise stays bounded by the three tests already in the ladder: reachability, the horizon ceiling (first-passage time grows with stop x target), and the cost fraction. Consequential fixes: the reachability floor was a macro keyed to the fixed ratio and is now BarrierMinReachPct(rr) evaluated per rung (a raised ratio has a lower break-even, so a fixed floor would be the wrong strictness); the detectability break-even likewise; PooledGate now writes and matches the ACTUAL ratio (TargetRR()) rather than the floor. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 16:16:33 -04:00
#define BARRIER_TARGET_RR_MIN 2.0
//+------------------------------------------------------------------+
//| Snap a raised ratio to a coarse shared ladder. PooledGate pools |
//| instruments only when their structural break-even matches, and |
//| continuous per-instrument ratios would never match, silently |
//| emptying the pool. Snaps DOWN and is floored at the policy min. |
//+------------------------------------------------------------------+
feat(geometry): 1:2 becomes a FLOOR the swing legs may raise, and the scan can no longer undercut it Two coupled changes, both from measurements in today's SP500 H4 log. 1. THE SCAN WAS OVERRIDING THE DERIVER ON THE WRONG OBJECTIVE. At 13:37:47 DeriveBarrierGeometry produced stop 1.21*ATR / target 2.41*ATR - break-even 33.3%. Thirty-seven seconds later the barrier-geometry scan adopted 2:2 - break-even 50.9% - because it carried 0.0143 nats of entry-time information against the configured pair's 0.0075, and cleared its family-wise gate. Information is not expectancy, and the scan says so itself; nothing checked what the adoption did to the operating point. It did this: the fitted thresholds immediately after read 38.8% win vs 50.9% break-even (-12.2pp) and 48.3% vs 50.9% (-2.6pp), where the earlier model on this instrument at a 1:2 geometry had fitted +1.8pp. The deriver applies the ratio as user RISK POLICY; a scan that can crown 1:1 makes two subsystems disagree about one geometry - the same split this file already fixed once for the clamped-horizon rule. The scan now enrols and crowns only pairings at or above the floor; sub-floor pairs are still scored and printed (marked 'r') so the choice stays auditable. This is NOT the min-RR rule removed on 2026-08-09 - that one guarded a rejection filter that no longer exists. 2. THE RATIO IS A FLOOR, NOT A CAP (user: "the ratio of 1:2 is a minimum that I want, but it should not cap to that if the average zigzag moves gives more room"). BARRIER_TARGET_RR -> BARRIER_TARGET_RR_MIN. ComputeBarrierHorizonBars already scanned ZigZag pivots for leg DURATION; it now harvests leg RANGE in the same pass - two properties of one object, so the horizon and the target describe the same legs instead of two windows. The per-rung ratio is the floor raised toward median-leg/stop, snapped DOWN to a coarse ladder (2.0/2.5/3.0/4.0/5.0). The ladder is coarse on purpose: PooledGate pools only instruments whose structural break-even matches, and continuous per-instrument ratios would never match and would silently empty the pool. A leg is the right yardstick precisely because it owes NOTHING to the barrier - sizing a target off travel measured over the barrier's own horizon is the circular loop that ran EURUSD/USDCAD away to 14-31*ATR in 2026-08-07. The raise stays bounded by the three tests already in the ladder: reachability, the horizon ceiling (first-passage time grows with stop x target), and the cost fraction. Consequential fixes: the reachability floor was a macro keyed to the fixed ratio and is now BarrierMinReachPct(rr) evaluated per rung (a raised ratio has a lower break-even, so a fixed floor would be the wrong strictness); the detectability break-even likewise; PooledGate now writes and matches the ACTUAL ratio (TargetRR()) rather than the floor. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 16:16:33 -04:00
double BarrierSnapRr(const double wanted)
{
if(wanted >= 5.0)
return 5.0;
if(wanted >= 4.0)
return 4.0;
if(wanted >= 3.0)
return 3.0;
if(wanted >= 2.5)
return 2.5;
fix(geometry): the ratio raise was reported against reachability, not bounded by it First clean derivation after 6f2def0 produced stop 1.22 / target 4.86 = 1:4 on SP500 H4, and the labels came out Buy 4.3% / Sell 6.1% / Neutral 89.6% - a 20.7:1 imbalance, against 3.4:1 at 1:2 and 1.5:1 at 1:1 on the identical 10601 excursions. That is the class-collapse regime, not a geometry. c3daded proposed ONE leg-implied ratio per rung and left the reachability floor to REJECT the rung, with a comment claiming the raise was "bounded" by it. Rejection is not a bound. The 5.09*ATR median leg proposed 1:4 at every stop, all seven rungs then missed their own floor (0.7%-9.6% against 12-20%), the no-rung-clears fallback fired and re-proposed the same 1:4 at the tightest rung - and printed a WARNING predicting exactly the rare positive class that followed. The system diagnosed itself correctly and had no authority to act on it: the same shape as the scan-vs-deriver split c3daded fixed one layer up, reintroduced one layer down. Why the full leg overshoots: a ZigZag leg is pivot-to-pivot travel and an entry is not a pivot - it lands inside the leg, with roughly half of it left on average. Sizing the target at the whole median leg asks the market to deliver, from an arbitrary bar, the entire move it usually makes between extremes. Rather than assume the half and hard-code a factor, the raise now steps DOWN the snap ladder (5 -> 4 -> 3 -> 2.5 -> 2) until the first-passage ladder says the target is actually reached, stopping at the 1:2 policy floor because that is risk policy and not a measurement. The legs still raise the ratio wherever the travel supports it; the market decides how far. Applied in BOTH the rung loop and the fallback branch - omitting the fallback is what actually shipped the 20.7:1 labels, since that branch runs precisely when no rung was reachable. Rung rows now print "[legs proposed 1:R, unreached]" so a walked-back raise is visible as one. Verified: BarrierStepDownRr is strictly decreasing and bottoms at the floor, so both loops terminate; rung-row StringFormat re-counted at 18 specifiers / 18 arguments. NOT COMPILED - user compiles in MetaEditor.
2026-08-19 17:04:32 -04:00
return BARRIER_TARGET_RR_MIN;
}
//+------------------------------------------------------------------+
//| The same ladder walked DOWNWARD, one rung per call. The raise |
//| from the swing legs is only a PROPOSAL; DeriveBarrierGeometry |
//| steps it down until the implied target is one the market |
//| measurably reaches. Strictly decreasing, so the caller loop ends. |
//+------------------------------------------------------------------+
fix(geometry): the ratio raise was reported against reachability, not bounded by it First clean derivation after 6f2def0 produced stop 1.22 / target 4.86 = 1:4 on SP500 H4, and the labels came out Buy 4.3% / Sell 6.1% / Neutral 89.6% - a 20.7:1 imbalance, against 3.4:1 at 1:2 and 1.5:1 at 1:1 on the identical 10601 excursions. That is the class-collapse regime, not a geometry. c3daded proposed ONE leg-implied ratio per rung and left the reachability floor to REJECT the rung, with a comment claiming the raise was "bounded" by it. Rejection is not a bound. The 5.09*ATR median leg proposed 1:4 at every stop, all seven rungs then missed their own floor (0.7%-9.6% against 12-20%), the no-rung-clears fallback fired and re-proposed the same 1:4 at the tightest rung - and printed a WARNING predicting exactly the rare positive class that followed. The system diagnosed itself correctly and had no authority to act on it: the same shape as the scan-vs-deriver split c3daded fixed one layer up, reintroduced one layer down. Why the full leg overshoots: a ZigZag leg is pivot-to-pivot travel and an entry is not a pivot - it lands inside the leg, with roughly half of it left on average. Sizing the target at the whole median leg asks the market to deliver, from an arbitrary bar, the entire move it usually makes between extremes. Rather than assume the half and hard-code a factor, the raise now steps DOWN the snap ladder (5 -> 4 -> 3 -> 2.5 -> 2) until the first-passage ladder says the target is actually reached, stopping at the 1:2 policy floor because that is risk policy and not a measurement. The legs still raise the ratio wherever the travel supports it; the market decides how far. Applied in BOTH the rung loop and the fallback branch - omitting the fallback is what actually shipped the 20.7:1 labels, since that branch runs precisely when no rung was reachable. Rung rows now print "[legs proposed 1:R, unreached]" so a walked-back raise is visible as one. Verified: BarrierStepDownRr is strictly decreasing and bottoms at the floor, so both loops terminate; rung-row StringFormat re-counted at 18 specifiers / 18 arguments. NOT COMPILED - user compiles in MetaEditor.
2026-08-19 17:04:32 -04:00
double BarrierStepDownRr(const double rr)
{
if(rr > 4.01)
return 4.0;
if(rr > 3.01)
return 3.0;
if(rr > 2.51)
return 2.5;
feat(geometry): 1:2 becomes a FLOOR the swing legs may raise, and the scan can no longer undercut it Two coupled changes, both from measurements in today's SP500 H4 log. 1. THE SCAN WAS OVERRIDING THE DERIVER ON THE WRONG OBJECTIVE. At 13:37:47 DeriveBarrierGeometry produced stop 1.21*ATR / target 2.41*ATR - break-even 33.3%. Thirty-seven seconds later the barrier-geometry scan adopted 2:2 - break-even 50.9% - because it carried 0.0143 nats of entry-time information against the configured pair's 0.0075, and cleared its family-wise gate. Information is not expectancy, and the scan says so itself; nothing checked what the adoption did to the operating point. It did this: the fitted thresholds immediately after read 38.8% win vs 50.9% break-even (-12.2pp) and 48.3% vs 50.9% (-2.6pp), where the earlier model on this instrument at a 1:2 geometry had fitted +1.8pp. The deriver applies the ratio as user RISK POLICY; a scan that can crown 1:1 makes two subsystems disagree about one geometry - the same split this file already fixed once for the clamped-horizon rule. The scan now enrols and crowns only pairings at or above the floor; sub-floor pairs are still scored and printed (marked 'r') so the choice stays auditable. This is NOT the min-RR rule removed on 2026-08-09 - that one guarded a rejection filter that no longer exists. 2. THE RATIO IS A FLOOR, NOT A CAP (user: "the ratio of 1:2 is a minimum that I want, but it should not cap to that if the average zigzag moves gives more room"). BARRIER_TARGET_RR -> BARRIER_TARGET_RR_MIN. ComputeBarrierHorizonBars already scanned ZigZag pivots for leg DURATION; it now harvests leg RANGE in the same pass - two properties of one object, so the horizon and the target describe the same legs instead of two windows. The per-rung ratio is the floor raised toward median-leg/stop, snapped DOWN to a coarse ladder (2.0/2.5/3.0/4.0/5.0). The ladder is coarse on purpose: PooledGate pools only instruments whose structural break-even matches, and continuous per-instrument ratios would never match and would silently empty the pool. A leg is the right yardstick precisely because it owes NOTHING to the barrier - sizing a target off travel measured over the barrier's own horizon is the circular loop that ran EURUSD/USDCAD away to 14-31*ATR in 2026-08-07. The raise stays bounded by the three tests already in the ladder: reachability, the horizon ceiling (first-passage time grows with stop x target), and the cost fraction. Consequential fixes: the reachability floor was a macro keyed to the fixed ratio and is now BarrierMinReachPct(rr) evaluated per rung (a raised ratio has a lower break-even, so a fixed floor would be the wrong strictness); the detectability break-even likewise; PooledGate now writes and matches the ACTUAL ratio (TargetRR()) rather than the floor. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 16:16:33 -04:00
return BARRIER_TARGET_RR_MIN;
}
//--- SCALE ladder for the stop quantile, walked WIDEST FIRST, taking the first rung whose implied target
//--- is still reached often enough to be a trainable class. Without the reachability test this landed on
//--- 6.66*ATR reached on 3.3% of bars - the model trained to predict something that never happened.
fix(geometry): the target was small BY CONSTRUCTION - ratio is now policy, scale is measured, ladder ceiling removed The derivation read the stop from q75 of ADVERSE travel and the target from q50 of FAVOURABLE travel. Over one horizon those distributions are broadly the same shape, so q75 > q50 MECHANICALLY - the target came out smaller than the stop no matter what the market did. SP500 H4 shipped stop 3.07 / target 1.70: a 0.55:1 payoff needing 64.3%. That was never a measurement, it was two mismatched constants. The reachability line printed beside it - "target on 50.0% of bars, stop on 25.0%" - is exactly 1-q50 and 1-q75. Tautological. It cannot disconfirm anything, and it read as validation. WIDTH AND RATIO ARE INDEPENDENT AND ONLY ONE PAYS. EV = edge x width; ratio is EV-neutral (a driftless walk reaches +m before -k with probability k/(k+m), which IS break-even). Width is what buys cost efficiency: the spread is a fixed 0.047*ATR here, so the shipped 4.77*ATR width paid it 21 times per unit of travel. So: RATIO = policy. BARRIER_TARGET_RR = 2.0 (user's 1:2). Break-even 33.3%. SCALE = measured. The stop quantile is chosen from a ladder, WIDEST FIRST, taking the first rung whose implied 2x target is still reached often enough to be a trainable class. That last clause is the difference from the min-reward:risk raise removed in 2026-08-09, which forced target = 2 x stop with NO reachability test, landed on 6.66*ATR reachable on 3.3% of bars, and trained the model to predict something that essentially never happened. Same ratio; the scale now retreats until the data says the target is attainable. Every rung is logged. LADDER CEILING REMOVED. BARRIER_LADDER stopped at 5.00 and the expectancy scan's "best resolvable pair on width alone" came back as stop 5.05 / target 4.95 - it pinned to the top rung. A recommendation landing exactly on the edge of its own search space is a boundary, not a finding: it cannot tell "5 ATR is optimal" from "5 ATR is all we allowed". Extended to 20*ATR (8 -> 14 rungs). Nothing else needs editing - every consumer is parameterised by BARRIER_LADDER_COUNT - and the horizon constraints (decided >= 60%, reachability floor) now bind instead of a constant. THE SCAN COULD NOT SEE THE SHIPPED GEOMETRY. ReportBarrierGeometryScan looked the configured pair up in its integer grid, and DeriveBarrierGeometry produces CONTINUOUS multiples (3.07/1.70) that can never equal a grid point - so cfgExcess stayed at its -1.0 sentinel and the report printed "configured 3:2 scores -1.00000", which reads as a catastrophic score and actually means "never evaluated". Worse, the grid skipped target<stop entirely because it "inverts the trade's whole premise" - while the derivation was shipping exactly that. The incumbent is now always scored as a peer (never crowned; it is already in force and is not an enum pairing the scan could adopt). BREAK-EVEN NOW INCLUDES THE SPREAD. Every report quoted the frictionless SL/(SL+TP). On SP500 H4 that read 64.3% while the MEASURED zero-skill rate was 62.1% - a 2.2pp gap that IS the cost, and that made every model look 2.2pp better than it was. CostAdjustedBreakEvenPct() prices a win at (TP - spread) and a loss at (SL + spread), matching the expectancy scan's convention exactly so the two reports cannot disagree. It also feeds FitDirConfThreshold, which is the correctness half: the operating point subtracts break-even from precision, so the frictionless figure made every candidate threshold look better by the width of the spread - 2.2pp against a measured edge of 2.3pp, i.e. very nearly all of it. Era line now carries both: "break-even 64.3% frictionless, 66.6% AFTER SPREAD". Forces a full relabel and retrain. Requested. NOT COMPILED - user compiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:34:32 -04:00
#define BARRIER_SL_QUANTILE_COUNT 7
const double BARRIER_SL_QUANTILE_LADDER[BARRIER_SL_QUANTILE_COUNT] =
{0.90, 0.85, 0.80, 0.75, 0.70, 0.60, 0.50};
//--- Fallback quantile when no rung clears the reachability floor.
fix: normalise the asymmetry target - the raw one is confounded by volatility Three symbols ran the excursion test. RANGE/UP/DOWN cleared on all three; raw ASYMMETRY cleared on EURUSD and USDCAD at p=0.0050 and not on SP500 (p=0.1045). That looked like the first directional signal this project has found. It probably is not, and the test as built could not tell. (up-dn) IS NOT SCALE-FREE. If sigma is predictable - and RANGE clears at ~4x its null on every instrument - and the directional part is symmetric noise eps, then up-dn ~ sigma*eps, so a large sigma pushes the value into BOTH outer terciles. A pure volatility predictor scores positive MI against a 3-bin (up-dn) while carrying no directional information at all. Crucially that confound REPLICATES, so reproducing on two instruments is not evidence against it - and the effect sizes fit it: asymmetry runs 1.3-1.6x its null where RANGE runs ~4x, and carries ~0.1% of the target's entropy against RANGE's ~0.9%. That is the shape of a leaked fraction of the volatility signal, not an independent one. So add (up-dn)/(up+dn): bounded in [-1,+1], volatility divided out, and the only target a directional claim may rest on. The verdict now separates the cases and NAMES the confound when raw clears while normalised does not, instead of reporting the raw line as a finding. Two bugs of mine in the same block, both caught by output rather than review: - The derived-geometry line had a MISORDERED argument list: it printed "stop 25.00*ATR (q3 of adverse travel)" - the quantile percentage as the multiple and the multiple as the quantile. Real values were 2.61 stop / 8.03 target. A 25*ATR stop is absurd on its face, which is why it was seen. - THE STOP QUANTILE WAS BACKWARDS, and this one changes labels. It was 0.25 "so ordinary noise does not reach it", but q25 means 75% of bars EXCEED the stop - hit three times in four. The printed reachability said exactly that ("stop on 75.0% of bars"). Now 0.75. A quantile is a threshold, not a rate. This is the entire reason reachability is measured and printed rather than assumed. Also raises BARRIER_DERIVE_MAX_PASSES 3 -> 5: SP500 did not settle in 3 (stop still moving ~14% per pass) while EURUSD and USDCAD converged on pass 2. And bounds both quantile indices with MathMin(..., n-1) so q=1.0 cannot run off the end of the sorted array. The geometry from the previous run is NOT usable and the asymmetry result is unresolved, not established. Both are decided by the next run. FORCES A FULL RETRAIN (the stop quantile changes every label). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 13:04:13 -04:00
#define BARRIER_SL_QUANTILE 0.75
feat(geometry): measure per-candidate barriers against the one global pair Stage 2a of the candidate-conditional geometry the record has named as next and never built. MEASUREMENT ONLY - no order uses it yet. WHY THIS AND NOT META-LABELING. Meta-labeling asks take-it-or-skip-it at fixed geometry, and its verdict stands: real skill, 0 operating points clearing break-even, and the 4070c5c retraction only moved that bar 1.4pp. The excursion head, by contrast, just cleared at 3.5-4.6 sigma on LSTM across four eras and beat the trailing-quantile incumbent. What is learnable here is MAGNITUDE, so the lever is the geometry, not the veto. A per-candidate rung means a per-candidate break-even, which a binary gate cannot express. HOW IT IS SCORED. At each OOS call the same bar is resolved under the incumbent pair AND under the pair this bar's excursion head would choose, and the paired difference is accumulated in R with a 2-sigma test. Both legs come from the SAME first-passage ladder - four array reads, no re-walk, exact even on the ~28% of bars where both barriers were touched. Mixing the ladder with the price walk here would measure the discrepancy between two of our own evaluators rather than the effect of the geometry, which is precisely what f8ac10c had to unpick one layer over. The candidate pair applies the GLOBAL derivation's own rule per bar: stop at BARRIER_SL_QUANTILE of adverse travel, target at the median of favourable. Neither creates expectancy; what moves is the break-even, which is why the report quotes R and never a win rate. FREE VALIDATION. The sigma gate predicts LSTM helps and CONV hurts. If the R difference reproduces that ordering across members, the head's usefulness is confirmed by a second, independent measurement. If it does not, something is wrong and this must not be wired to orders. Two things caught while writing it, both silent if missed: - The ladder stores TRAVEL FROM ENTRY, and the scan's mapping is risk = ladder + spread but reward = ladder - spread, so the two legs convert with OPPOSITE signs. The stop leg had the sign backwards. - A GEOMETRY_BUDGET_MS wall clock, because this adds a feature-window build and a head forward per OOS call to a walk that already runs unchunked at era end on a single-threaded EA. That is the shape that got the process force- terminated on 2026-08-21. It stops scoring, never the replay, and the report prints how many calls it covered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 08:29:59 -04:00
//--- ...and the target at the MEDIAN of favourable travel, so it is reached about half the time by
//--- construction. The global derivation applies both once per era; CandidateGeometryFor applies the
//--- same two per bar. Neither creates expectancy - what moves per candidate is the break-even.
#define BARRIER_TP_QUANTILE 0.50
//--- Milliseconds the candidate-geometry measurement may spend inside one era's replay. A diagnostic
//--- that freezes the chart is worse than a missing diagnostic; the report prints its own coverage.
#define GEOMETRY_BUDGET_MS 5000
//--- Below this many resolved excursions the quantiles are too noisy to key a training target on.
feat: derive the ATR multiples from measured excursions - no hardcoded geometry The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in 3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of eleven guesses is not deriving anything. WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It ranks pairings by how predictable their OUTCOME is - a question about direction. The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x. Hence the scan failing its own gate on every run, and its "winner" wandering 2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is strongly measurable, so derive the geometry from that instead. stop = q25 of measured ADVERSE travel (ordinary noise does not reach it) target = q50 of measured FAVOURABLE travel (reached ~half the time, by construction, inside the horizon) Continuous, in ATR units, superseding the enum multiples. Reachability ("target on X% of bars, stop on Y%") and the implied break-even are printed so the choice is auditable rather than trusted. FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the horizon with the target (first-passage time grows with the band) and the excursions are measured OVER the horizon, so target -> horizon -> excursions -> target is a real loop - deriving once sizes the target from travel measured under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at 3 passes, and says so if it does not settle. Does NOT create expectancy, and the log says as much: chance precision equals break-even at every geometry (m/(m+k) on both sides). It buys a target the market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio forces a target the market rarely reaches, it WARNS rather than overriding - the ratio is the user's risk policy, so the honest move is to state its cost. That is the collision that once rejected 100% of setups. Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg files written earlier today still load (their length guard finds no doubles) and a model that carries them was trained on them and never re-derives. Also fixes a message from e5ceed6 that claimed "this model resumed from disk" unconditionally - it printed above a "seeding era 0" line on a brand-new model, because the branch fires whenever the cache is not built, which is equally true before a fresh model's first prebuild. A diagnostic that misreports its own trigger is worse than one that says nothing: it gets quoted back as evidence. FORCES A FULL RETRAIN (labels change). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 12:06:25 -04:00
#define BARRIER_DERIVE_MIN_SAMPLES 500
//--- FIRST-PASSAGE LADDER. Recording first-touch AGE per rung makes any pair evaluable exactly,
//--- with no re-walk. Barrier prices would need four ladders and bake in today's spread.
fix(geometry): the target was small BY CONSTRUCTION - ratio is now policy, scale is measured, ladder ceiling removed The derivation read the stop from q75 of ADVERSE travel and the target from q50 of FAVOURABLE travel. Over one horizon those distributions are broadly the same shape, so q75 > q50 MECHANICALLY - the target came out smaller than the stop no matter what the market did. SP500 H4 shipped stop 3.07 / target 1.70: a 0.55:1 payoff needing 64.3%. That was never a measurement, it was two mismatched constants. The reachability line printed beside it - "target on 50.0% of bars, stop on 25.0%" - is exactly 1-q50 and 1-q75. Tautological. It cannot disconfirm anything, and it read as validation. WIDTH AND RATIO ARE INDEPENDENT AND ONLY ONE PAYS. EV = edge x width; ratio is EV-neutral (a driftless walk reaches +m before -k with probability k/(k+m), which IS break-even). Width is what buys cost efficiency: the spread is a fixed 0.047*ATR here, so the shipped 4.77*ATR width paid it 21 times per unit of travel. So: RATIO = policy. BARRIER_TARGET_RR = 2.0 (user's 1:2). Break-even 33.3%. SCALE = measured. The stop quantile is chosen from a ladder, WIDEST FIRST, taking the first rung whose implied 2x target is still reached often enough to be a trainable class. That last clause is the difference from the min-reward:risk raise removed in 2026-08-09, which forced target = 2 x stop with NO reachability test, landed on 6.66*ATR reachable on 3.3% of bars, and trained the model to predict something that essentially never happened. Same ratio; the scale now retreats until the data says the target is attainable. Every rung is logged. LADDER CEILING REMOVED. BARRIER_LADDER stopped at 5.00 and the expectancy scan's "best resolvable pair on width alone" came back as stop 5.05 / target 4.95 - it pinned to the top rung. A recommendation landing exactly on the edge of its own search space is a boundary, not a finding: it cannot tell "5 ATR is optimal" from "5 ATR is all we allowed". Extended to 20*ATR (8 -> 14 rungs). Nothing else needs editing - every consumer is parameterised by BARRIER_LADDER_COUNT - and the horizon constraints (decided >= 60%, reachability floor) now bind instead of a constant. THE SCAN COULD NOT SEE THE SHIPPED GEOMETRY. ReportBarrierGeometryScan looked the configured pair up in its integer grid, and DeriveBarrierGeometry produces CONTINUOUS multiples (3.07/1.70) that can never equal a grid point - so cfgExcess stayed at its -1.0 sentinel and the report printed "configured 3:2 scores -1.00000", which reads as a catastrophic score and actually means "never evaluated". Worse, the grid skipped target<stop entirely because it "inverts the trade's whole premise" - while the derivation was shipping exactly that. The incumbent is now always scored as a peer (never crowned; it is already in force and is not an enum pairing the scan could adopt). BREAK-EVEN NOW INCLUDES THE SPREAD. Every report quoted the frictionless SL/(SL+TP). On SP500 H4 that read 64.3% while the MEASURED zero-skill rate was 62.1% - a 2.2pp gap that IS the cost, and that made every model look 2.2pp better than it was. CostAdjustedBreakEvenPct() prices a win at (TP - spread) and a loss at (SL + spread), matching the expectancy scan's convention exactly so the two reports cannot disagree. It also feeds FitDirConfThreshold, which is the correctness half: the operating point subtracts break-even from precision, so the frictionless figure made every candidate threshold look better by the width of the spread - 2.2pp against a measured edge of 2.3pp, i.e. very nearly all of it. Era line now carries both: "break-even 64.3% frictionless, 66.6% AFTER SPREAD". Forces a full relabel and retrain. Requested. NOT COMPILED - user compiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:34:32 -04:00
#define BARRIER_LADDER_COUNT 14
const double BARRIER_LADDER[BARRIER_LADDER_COUNT] =
{0.50, 0.75, 1.00, 1.50, 2.00, 3.00, 4.00, 5.00, 6.50, 8.00, 10.00, 13.00, 16.00, 20.00};
//--- The derivation is a FIXED-POINT ITERATION: the horizon scales with the target, and the excursions
//--- are measured OVER that horizon, so target -> horizon -> excursions -> target is a loop. Deriving
//--- once would set the target from travel measured under the OLD horizon.
fix: normalise the asymmetry target - the raw one is confounded by volatility Three symbols ran the excursion test. RANGE/UP/DOWN cleared on all three; raw ASYMMETRY cleared on EURUSD and USDCAD at p=0.0050 and not on SP500 (p=0.1045). That looked like the first directional signal this project has found. It probably is not, and the test as built could not tell. (up-dn) IS NOT SCALE-FREE. If sigma is predictable - and RANGE clears at ~4x its null on every instrument - and the directional part is symmetric noise eps, then up-dn ~ sigma*eps, so a large sigma pushes the value into BOTH outer terciles. A pure volatility predictor scores positive MI against a 3-bin (up-dn) while carrying no directional information at all. Crucially that confound REPLICATES, so reproducing on two instruments is not evidence against it - and the effect sizes fit it: asymmetry runs 1.3-1.6x its null where RANGE runs ~4x, and carries ~0.1% of the target's entropy against RANGE's ~0.9%. That is the shape of a leaked fraction of the volatility signal, not an independent one. So add (up-dn)/(up+dn): bounded in [-1,+1], volatility divided out, and the only target a directional claim may rest on. The verdict now separates the cases and NAMES the confound when raw clears while normalised does not, instead of reporting the raw line as a finding. Two bugs of mine in the same block, both caught by output rather than review: - The derived-geometry line had a MISORDERED argument list: it printed "stop 25.00*ATR (q3 of adverse travel)" - the quantile percentage as the multiple and the multiple as the quantile. Real values were 2.61 stop / 8.03 target. A 25*ATR stop is absurd on its face, which is why it was seen. - THE STOP QUANTILE WAS BACKWARDS, and this one changes labels. It was 0.25 "so ordinary noise does not reach it", but q25 means 75% of bars EXCEED the stop - hit three times in four. The printed reachability said exactly that ("stop on 75.0% of bars"). Now 0.75. A quantile is a threshold, not a rate. This is the entire reason reachability is measured and printed rather than assumed. Also raises BARRIER_DERIVE_MAX_PASSES 3 -> 5: SP500 did not settle in 3 (stop still moving ~14% per pass) while EURUSD and USDCAD converged on pass 2. And bounds both quantile indices with MathMin(..., n-1) so q=1.0 cannot run off the end of the sorted array. The geometry from the previous run is NOT usable and the asymmetry result is unresolved, not established. Both are decided by the next run. FORCES A FULL RETRAIN (the stop quantile changes every label). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 13:04:13 -04:00
#define BARRIER_DERIVE_MAX_PASSES 5
feat: derive the ATR multiples from measured excursions - no hardcoded geometry The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in 3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of eleven guesses is not deriving anything. WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It ranks pairings by how predictable their OUTCOME is - a question about direction. The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x. Hence the scan failing its own gate on every run, and its "winner" wandering 2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is strongly measurable, so derive the geometry from that instead. stop = q25 of measured ADVERSE travel (ordinary noise does not reach it) target = q50 of measured FAVOURABLE travel (reached ~half the time, by construction, inside the horizon) Continuous, in ATR units, superseding the enum multiples. Reachability ("target on X% of bars, stop on Y%") and the implied break-even are printed so the choice is auditable rather than trusted. FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the horizon with the target (first-passage time grows with the band) and the excursions are measured OVER the horizon, so target -> horizon -> excursions -> target is a real loop - deriving once sizes the target from travel measured under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at 3 passes, and says so if it does not settle. Does NOT create expectancy, and the log says as much: chance precision equals break-even at every geometry (m/(m+k) on both sides). It buys a target the market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio forces a target the market rarely reaches, it WARNS rather than overriding - the ratio is the user's risk policy, so the honest move is to state its cost. That is the collision that once rejected 100% of setups. Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg files written earlier today still load (their length guard finds no doubles) and a model that carries them was trained on them and never re-derives. Also fixes a message from e5ceed6 that claimed "this model resumed from disk" unconditionally - it printed above a "seeding era 0" line on a brand-new model, because the branch fires whenever the cache is not built, which is equally true before a fresh model's first prebuild. A diagnostic that misreports its own trigger is worse than one that says nothing: it gets quoted back as evidence. FORCES A FULL RETRAIN (labels change). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 12:06:25 -04:00
#define BARRIER_DERIVE_TOLERANCE 0.05
//--- Reachability floor as a FRACTION OF BREAK-EVEN, not an absolute percentage: break-even for a
//--- 1:RR trade is 100/(1+RR), so an absolute 20% is 0.60x break-even at RR=2 but 0.80x at RR=3 -
//--- tightening the test simply because the user asked for a bigger target.
fix(geometry): the reachability floor measured the WRONG WINDOW - my bug from bc57aca, and it cost real width RECONCILED: the derivation reported "target reached on 17.7% of bars" while the label cache reported Buy on 35.9%. Nothing was broken. They measure different windows, and both are correct: EXCURSION window ~12 bars (the SWING MEDIAN) - what m_excUpCache accumulates over. Deliberately short: sizing a barrier off travel measured over a horizon that itself scales with the barrier is circular, and it ran away to 14-31*ATR on EURUSD/USDCAD in 2026-08-07. That guard is correct and stays. BARRIER horizon 64 bars - what the LABEL walk and the first-passage ladder run over, and how long the EA actually holds the trade. So `up >= target` is a 12-bar question and `label == Buy` is a 64-bar one, and the second can freely exceed the first. TripleBarrierLabel gates the excursion accumulation on `idx - t <= excWindow` while the barrier walk and the ladder run the full horizon - the split is explicit and intentional. THE BUG IS MINE. bc57aca's scale ladder tested reachability with `up[i] >= tp`, i.e. it asked the 12-bar question about a 64-bar trade. That understates reachability by ~2x, which is why EVERY wide rung was rejected and the geometry fell back to the tightest rung at 1.61/3.21. The data supported considerably wider; the test was just asking the wrong question. FIX: LadderWinShare() reads the answer off the first-passage ladder - target touched strictly before the stop, over the full horizon, tie to the stop. That is the identical question the label walk asks, so the ladder share and the Buy rate should now agree to within rung discretisation. Both legs snap to the SMALLEST rung at or above the requested multiple (harder target, harder stop) so the floor stays conservative. Expect the scale ladder to select a WIDER rung on the next relabel. On this data the excursion test read 17.7% at q50 where the true full-horizon share is 35.9%, so rungs that scored 8.1% and 2.8% were likely well above the floor. ALSO: - Window reconciliation now PRINTED every derivation: excursion travel share, ladder win share, and the label cache's Buy share side by side, with the ladder-vs-label gap flagged if it exceeds rung discretisation. Those two must agree; if they ever stop agreeing, one of them is wrong and the line says so. - Renamed tpReach/slReach -> tpTravel/slTravel and relabelled the log line. They describe the EXCURSION window and are near-tautological there (a q50 stop is exceeded by ~50% of bars); calling them "reached within the horizon" is what made the two quantities look like one. - BARRIER_MIN_TP_REACH_PCT is now BARRIER_MIN_REACH_FRACTION_OF_BE (0.60) x break-even instead of a hardcoded 20.0. Break-even for 1:RR is 100/(1+RR), so the absolute floor silently tightened as RR rose - 0.60x at RR=2 but 0.80x at RR=3, penalising the user for asking for a bigger target. Evaluates to exactly 20.0% at the shipped RR=2, so this is a no-op today and correct if the knob moves. NOT COMPILED - user compiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:51:08 -04:00
#define BARRIER_MIN_REACH_FRACTION_OF_BE 0.60
//+------------------------------------------------------------------+
//| Reachability floor for one rung. A FUNCTION of the ratio, not a |
//| constant, because the ratio is now per-rung - a floor computed |
//| from one fixed RR would be the wrong strictness for every rung |
//| the swing legs raised. |
//+------------------------------------------------------------------+
feat(geometry): 1:2 becomes a FLOOR the swing legs may raise, and the scan can no longer undercut it Two coupled changes, both from measurements in today's SP500 H4 log. 1. THE SCAN WAS OVERRIDING THE DERIVER ON THE WRONG OBJECTIVE. At 13:37:47 DeriveBarrierGeometry produced stop 1.21*ATR / target 2.41*ATR - break-even 33.3%. Thirty-seven seconds later the barrier-geometry scan adopted 2:2 - break-even 50.9% - because it carried 0.0143 nats of entry-time information against the configured pair's 0.0075, and cleared its family-wise gate. Information is not expectancy, and the scan says so itself; nothing checked what the adoption did to the operating point. It did this: the fitted thresholds immediately after read 38.8% win vs 50.9% break-even (-12.2pp) and 48.3% vs 50.9% (-2.6pp), where the earlier model on this instrument at a 1:2 geometry had fitted +1.8pp. The deriver applies the ratio as user RISK POLICY; a scan that can crown 1:1 makes two subsystems disagree about one geometry - the same split this file already fixed once for the clamped-horizon rule. The scan now enrols and crowns only pairings at or above the floor; sub-floor pairs are still scored and printed (marked 'r') so the choice stays auditable. This is NOT the min-RR rule removed on 2026-08-09 - that one guarded a rejection filter that no longer exists. 2. THE RATIO IS A FLOOR, NOT A CAP (user: "the ratio of 1:2 is a minimum that I want, but it should not cap to that if the average zigzag moves gives more room"). BARRIER_TARGET_RR -> BARRIER_TARGET_RR_MIN. ComputeBarrierHorizonBars already scanned ZigZag pivots for leg DURATION; it now harvests leg RANGE in the same pass - two properties of one object, so the horizon and the target describe the same legs instead of two windows. The per-rung ratio is the floor raised toward median-leg/stop, snapped DOWN to a coarse ladder (2.0/2.5/3.0/4.0/5.0). The ladder is coarse on purpose: PooledGate pools only instruments whose structural break-even matches, and continuous per-instrument ratios would never match and would silently empty the pool. A leg is the right yardstick precisely because it owes NOTHING to the barrier - sizing a target off travel measured over the barrier's own horizon is the circular loop that ran EURUSD/USDCAD away to 14-31*ATR in 2026-08-07. The raise stays bounded by the three tests already in the ladder: reachability, the horizon ceiling (first-passage time grows with stop x target), and the cost fraction. Consequential fixes: the reachability floor was a macro keyed to the fixed ratio and is now BarrierMinReachPct(rr) evaluated per rung (a raised ratio has a lower break-even, so a fixed floor would be the wrong strictness); the detectability break-even likewise; PooledGate now writes and matches the ACTUAL ratio (TargetRR()) rather than the floor. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 16:16:33 -04:00
double BarrierMinReachPct(const double rr)
{
return BARRIER_MIN_REACH_FRACTION_OF_BE * 100.0 / (1.0 + rr);
}
//--- WHICH END OF THE SCALE LADDER WINS. The two objectives are genuinely opposed, each correct in
//--- its own phase. WIDE is right once an edge is KNOWN - EV = edge x width against a fixed spread.
feat(measurement): fix zero-skill denominator, publish the deploy bar, measure lifespan per rung, add a MEASURE scale objective The last run could not have demonstrated an edge either way, and nothing in the log said so. Four changes so it does. 1. THE ZERO-SKILL LINE DIVIDED BY THE WRONG DENOMINATOR. m_oosWinLongTotal resets every era; m_oosSamples only resets on a full model reset. So 'always-long %' decayed as ~1/era: a run whose true rate is 37% printed 1.2% at era 33 and 0.0% at era 2219. This is the SAME bug already found and fixed for logBuyPredPct thirty lines above ('era-15 Buy:2% that was really ~30%'), left in the one line whose whole job is to be the reference every other number is read against. Correct at era 1, wrong everywhere after - including the '62% zero-skill' figure in the 2026-08-16 notes. Now per-era, and always-short is finally readable. 2. THE DEPLOY GATE STATES ITS OWN BAR. 'edge -1pp' era after era cannot separate 'short by a hair' from 'short by an amount no strategy could cover'. The era line now prints the required win rate, the SE, the effective n and the lifespan it was deflated by; above 100% it says UNREACHABLE. At 4,738 OOS bars and L=75.6 there are ~63 independent observations, putting the bar near 66% at typical coverage. 3. LIFESPAN MEASURED PER RUNG. The first-passage cache already stores touch ages at every ladder level, so each candidate geometry's resolution time is readable without training on it - L-vs-width becomes a measurement across the whole ladder in ONE run rather than a second chart. Each rung reports L, n_eff, min provable edge and min provable EV. 4. SCALE OBJECTIVE IS PHASE-AWARE, defaulting to MEASURE. Width and detectability are opposed: labels overlap by L, L grows like m*k = width^2 at fixed ratio, so min provable EV ~ width^2 while the cost saving from width is only linear. Doubling width quadruples the smallest EV you can prove. DEPLOY (widest that clears reachability) is right once an edge is known; MEASURE (narrowest that keeps round-trip spread under BARRIER_MAX_COST_FRACTION_PCT) is right while it still has to be shown. The direction does not depend on the exponent, and item 3 makes the exponent checkable. Fixed in review: m_lastRungLifespan is cleared on every LadderWinShare entry or a rejected rung reports the previous rung's lifespan as its own; per-rung detectability is labelled IS-sample based (the deriver may not see the holdout), so absolute figures are optimistic while the ranking is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:37:05 -04:00
#define BARRIER_SCALE_MEASURE 0 // narrowest rung that stays cost-efficient - maximises detectability
#define BARRIER_SCALE_DEPLOY 1 // widest rung that clears reachability - maximises EV per trade
#define BARRIER_SCALE_OBJECTIVE BARRIER_SCALE_MEASURE
//--- Stops MEASURE mode running to the tightest rung, mirroring the reachability floor that stops DEPLOY
//--- mode running to the widest. Round-trip cost is 2*spread; at the measured 0.047*ATR that admits any
//--- width down to ~3.1*ATR.
feat(measurement): fix zero-skill denominator, publish the deploy bar, measure lifespan per rung, add a MEASURE scale objective The last run could not have demonstrated an edge either way, and nothing in the log said so. Four changes so it does. 1. THE ZERO-SKILL LINE DIVIDED BY THE WRONG DENOMINATOR. m_oosWinLongTotal resets every era; m_oosSamples only resets on a full model reset. So 'always-long %' decayed as ~1/era: a run whose true rate is 37% printed 1.2% at era 33 and 0.0% at era 2219. This is the SAME bug already found and fixed for logBuyPredPct thirty lines above ('era-15 Buy:2% that was really ~30%'), left in the one line whose whole job is to be the reference every other number is read against. Correct at era 1, wrong everywhere after - including the '62% zero-skill' figure in the 2026-08-16 notes. Now per-era, and always-short is finally readable. 2. THE DEPLOY GATE STATES ITS OWN BAR. 'edge -1pp' era after era cannot separate 'short by a hair' from 'short by an amount no strategy could cover'. The era line now prints the required win rate, the SE, the effective n and the lifespan it was deflated by; above 100% it says UNREACHABLE. At 4,738 OOS bars and L=75.6 there are ~63 independent observations, putting the bar near 66% at typical coverage. 3. LIFESPAN MEASURED PER RUNG. The first-passage cache already stores touch ages at every ladder level, so each candidate geometry's resolution time is readable without training on it - L-vs-width becomes a measurement across the whole ladder in ONE run rather than a second chart. Each rung reports L, n_eff, min provable edge and min provable EV. 4. SCALE OBJECTIVE IS PHASE-AWARE, defaulting to MEASURE. Width and detectability are opposed: labels overlap by L, L grows like m*k = width^2 at fixed ratio, so min provable EV ~ width^2 while the cost saving from width is only linear. Doubling width quadruples the smallest EV you can prove. DEPLOY (widest that clears reachability) is right once an edge is known; MEASURE (narrowest that keeps round-trip spread under BARRIER_MAX_COST_FRACTION_PCT) is right while it still has to be shown. The direction does not depend on the exponent, and item 3 makes the exponent checkable. Fixed in review: m_lastRungLifespan is cleared on every LadderWinShare entry or a rejected rung reports the previous rung's lifespan as its own; per-rung detectability is labelled IS-sample based (the deriver may not see the holdout), so absolute figures are optimistic while the ranking is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:37:05 -04:00
#define BARRIER_MAX_COST_FRACTION_PCT 3.0
diag: is "optimal SL/TP" learnable? Score the features against excursions Proposed direction: train the net to predict entry/SL/TP that maximise return and minimise drawdown, rather than to classify direction. Before rebuilding a head, measure whether the target is learnable at all. That question splits into two that behave nothing alike: HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility clustering is about the most robust regularity in markets. WHICH WAY it goes first (the asymmetry) - direction, which is what every noise-floor verdict in this project has been about. Expectancy comes ONLY from the second. The first buys position sizing and drawdown control - worth having under prop-firm limits, but not an edge: exit management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with expectancy FLAT. Crucially this is NOT already answered. Every MI figure here scored the triple-barrier label, i.e. one specific question at one fixed geometry. A noise-floor result there says nothing about whether excursion MAGNITUDE is learnable - different target, different answer. Four targets, and the verdict is the CONTRAST, printed explicitly because the dangerous misreading of "UP clears" is "we can predict profitable trades": RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that SHOULD clear. Every prior verdict here lacked a control expected to pass; a range target at the floor indicts the measurement, not the market. UP / DOWN - MFE / MAE. ASYMMETRY - up-dn, the only one that can pay. Collected inside the walk the label already does (one max, one min per bar). The early-out when both barriers resolved is GONE: it would have truncated the excursions at whichever bar tripped the last barrier, making the measurement a function of the CURRENT SL/TP - the circularity this is trying to escape. The loop was already bounded by the horizon, so only the average cost moves. Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is fat-tailed and fixed-width bins would put nearly every row in bin 0; it also pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and to the barrier label's ~1.02 instead of confounded by class balance. Two bugs fixed in this code before it ever ran, both of which would have produced a plausible quiet wrong answer rather than an error: - TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the accumulators were reset, so one bar's excursions would be cached under another bar's index. Cleared at the top now, ahead of every return. - An unresolvable bar is still flagged as labelled but carries excursions of exactly 0. Under equal-frequency binning a block of identical zeros drags the lowest cut onto zero and a third of the sample lands in one uninformative bin - a depressed score that reads as "not predictable", a false negative in the direction that would wrongly kill the idea. Rows where both excursions are zero are dropped; price cannot travel zero both ways over a whole horizon. Read-only diagnostic. No topology or label change: no retrain of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 10:22:41 -04:00
#define MI_TARGET_BARRIER 0 // shipped 3-class triple-barrier label
#define MI_TARGET_EXC_UP 1 // (maxHigh - entry)/ATR over the horizon, 3 equal-frequency bins
#define MI_TARGET_EXC_DOWN 2 // (entry - minLow)/ATR
#define MI_TARGET_EXC_RANGE 3 // up + down: pure realised volatility, the control that SHOULD clear
fix: normalise the asymmetry target - the raw one is confounded by volatility Three symbols ran the excursion test. RANGE/UP/DOWN cleared on all three; raw ASYMMETRY cleared on EURUSD and USDCAD at p=0.0050 and not on SP500 (p=0.1045). That looked like the first directional signal this project has found. It probably is not, and the test as built could not tell. (up-dn) IS NOT SCALE-FREE. If sigma is predictable - and RANGE clears at ~4x its null on every instrument - and the directional part is symmetric noise eps, then up-dn ~ sigma*eps, so a large sigma pushes the value into BOTH outer terciles. A pure volatility predictor scores positive MI against a 3-bin (up-dn) while carrying no directional information at all. Crucially that confound REPLICATES, so reproducing on two instruments is not evidence against it - and the effect sizes fit it: asymmetry runs 1.3-1.6x its null where RANGE runs ~4x, and carries ~0.1% of the target's entropy against RANGE's ~0.9%. That is the shape of a leaked fraction of the volatility signal, not an independent one. So add (up-dn)/(up+dn): bounded in [-1,+1], volatility divided out, and the only target a directional claim may rest on. The verdict now separates the cases and NAMES the confound when raw clears while normalised does not, instead of reporting the raw line as a finding. Two bugs of mine in the same block, both caught by output rather than review: - The derived-geometry line had a MISORDERED argument list: it printed "stop 25.00*ATR (q3 of adverse travel)" - the quantile percentage as the multiple and the multiple as the quantile. Real values were 2.61 stop / 8.03 target. A 25*ATR stop is absurd on its face, which is why it was seen. - THE STOP QUANTILE WAS BACKWARDS, and this one changes labels. It was 0.25 "so ordinary noise does not reach it", but q25 means 75% of bars EXCEED the stop - hit three times in four. The printed reachability said exactly that ("stop on 75.0% of bars"). Now 0.75. A quantile is a threshold, not a rate. This is the entire reason reachability is measured and printed rather than assumed. Also raises BARRIER_DERIVE_MAX_PASSES 3 -> 5: SP500 did not settle in 3 (stop still moving ~14% per pass) while EURUSD and USDCAD converged on pass 2. And bounds both quantile indices with MathMin(..., n-1) so q=1.0 cannot run off the end of the sorted array. The geometry from the previous run is NOT usable and the asymmetry result is unresolved, not established. Both are decided by the next run. FORCES A FULL RETRAIN (the stop quantile changes every label). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 13:04:13 -04:00
#define MI_TARGET_EXC_ASYM 4 // up - down: RAW asymmetry - CONFOUNDED BY VOLATILITY, see below
//--- SCALE-FREE asymmetry, and the only one of the two that can support a directional claim.
//--- Dividing by (up+dn) leaves the question actually being asked: given that price moved, which
//--- way.
fix: normalise the asymmetry target - the raw one is confounded by volatility Three symbols ran the excursion test. RANGE/UP/DOWN cleared on all three; raw ASYMMETRY cleared on EURUSD and USDCAD at p=0.0050 and not on SP500 (p=0.1045). That looked like the first directional signal this project has found. It probably is not, and the test as built could not tell. (up-dn) IS NOT SCALE-FREE. If sigma is predictable - and RANGE clears at ~4x its null on every instrument - and the directional part is symmetric noise eps, then up-dn ~ sigma*eps, so a large sigma pushes the value into BOTH outer terciles. A pure volatility predictor scores positive MI against a 3-bin (up-dn) while carrying no directional information at all. Crucially that confound REPLICATES, so reproducing on two instruments is not evidence against it - and the effect sizes fit it: asymmetry runs 1.3-1.6x its null where RANGE runs ~4x, and carries ~0.1% of the target's entropy against RANGE's ~0.9%. That is the shape of a leaked fraction of the volatility signal, not an independent one. So add (up-dn)/(up+dn): bounded in [-1,+1], volatility divided out, and the only target a directional claim may rest on. The verdict now separates the cases and NAMES the confound when raw clears while normalised does not, instead of reporting the raw line as a finding. Two bugs of mine in the same block, both caught by output rather than review: - The derived-geometry line had a MISORDERED argument list: it printed "stop 25.00*ATR (q3 of adverse travel)" - the quantile percentage as the multiple and the multiple as the quantile. Real values were 2.61 stop / 8.03 target. A 25*ATR stop is absurd on its face, which is why it was seen. - THE STOP QUANTILE WAS BACKWARDS, and this one changes labels. It was 0.25 "so ordinary noise does not reach it", but q25 means 75% of bars EXCEED the stop - hit three times in four. The printed reachability said exactly that ("stop on 75.0% of bars"). Now 0.75. A quantile is a threshold, not a rate. This is the entire reason reachability is measured and printed rather than assumed. Also raises BARRIER_DERIVE_MAX_PASSES 3 -> 5: SP500 did not settle in 3 (stop still moving ~14% per pass) while EURUSD and USDCAD converged on pass 2. And bounds both quantile indices with MathMin(..., n-1) so q=1.0 cannot run off the end of the sorted array. The geometry from the previous run is NOT usable and the asymmetry result is unresolved, not established. Both are decided by the next run. FORCES A FULL RETRAIN (the stop quantile changes every label). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 13:04:13 -04:00
#define MI_TARGET_EXC_ASYM_NORM 5
//--- Significance the tuner's winner must reach AFTER correcting for best-of-N. This selector overwrites
//--- the user's indicator settings and forces a fresh topology, so a gate has to exist.
fix: make the indicator tuner actually measure, and gate what it installs ROOT CAUSE of the zero spread measured on SP500 H1 2026-08-07 (all 17 candidates returned exactly 0.00359 nats): the tune loop re-inits the indicators and then scores, with no RefreshData() between. ReInitADIndicators() does its part - Create() builds a NEW handle carrying the new parameters, and the feature cache is flagged stale so features really are recomputed. But BufferTempDataCompute() reads the CIndicatorBuffer objects, and only Refresh() copies data out of a handle into those. So every candidate was scored on values still held from the PREVIOUS handle. My earlier guess in the diagnostic ("suspect the feature cache") was wrong: the cache invalidation works. Two things land together, because neither is safe alone: 1. RefreshData() after the re-init, so a candidate is scored on its own features. 2. A SELECTION GATE on the install. bestScore is a MAXIMUM over candidates, and the maximum of N draws from a null beats its incumbent almost every time - so "it beat the incumbent" installs noise. This selector is the highest-stakes of the three found in this audit because it ACTS: it overwrites the user's configured indicator settings and forces BuildFreshTopology(), so the network then trains on whatever the noise picked. Fixing (1) without (2) would have made a dormant bug actively harmful. The gate draws the winner's own permutation null once, then corrects the p-value for having chosen it out of N with Sidak: p_family = 1 - (1-p)^N. Sidak rather than the max-of-N resample used by the geometry scan because each candidate here has a DIFFERENT feature set, so their draws cannot be pooled; Sidak needs only the one null. Exact under independence, mildly anti-conservative under positive dependence - stated in the comment rather than hidden. A rejected winner restores the configured settings, which best[] cannot do since the descent mutates it. Also reports the least-ready tunable handle's BarsCalculated(). IndicatorCreate() calculates asynchronously, so if the spread is STILL zero the handles simply are not done and the tuner needs to yield between candidates rather than score them back to back - a state machine like the label prebuild. That distinction is now readable from the log instead of requiring another guess. No input, topology or label change: no retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 09:31:06 -04:00
#define MI_TUNE_ALPHA 0.05
//--- WHAT THE TUNER OPTIMISES FOR. RANGE is the one target with measured signal (4x its null,
//--- p=0.005, with a working positive control) and it is what the excursion head is trained to
//--- predict.
#define MI_TUNE_TARGET MI_TARGET_EXC_RANGE
//--- Ceiling on profiled lags, sizing the retained-draw matrix. A larger m_historyBars is covered to the
//--- first MI_LAG_MAX_PROFILE-1 lags and says so via the lag count it prints.
fix: correct the lag profile across lags too - it contradicted itself 3271f1e tested each of ~21 lags against its OWN null at alpha 0.05 and starred whatever cleared. That is about one false positive per run before any signal exists, and because neighbouring lags share nearly their entire feature window the false positives arrive in CLUSTERS that read like a hump. It did exactly that on SP500 H1, twice in one afternoon on identical data: 13:55 nothing clears at any lag headline MI p=0.4478 16:22 k6/k10/k12/k16 starred, headline MI p=0.8756, observed "information survives to lag 16" BELOW its own null mean Same 31 features, same 2009 samples, same 287 blocks, cross-asset absent in both - so this was not two different measurements. Non-replication on identical data is the signature of an uncorrected multiple comparison, and acting on the second run would have pinned the lookback to 17 off noise. Galling detail: 04ee2e1 had just added exactly this correction to the barrier-geometry scan one function below. The rigorous bar went on the report with 6 candidates and the naive one stayed on the report with 21. So the lag profile now uses the same construction as the geometry winner test: one draw from every lag, keep the largest, repeat; a lag clears only by beating that distribution. Draws centred leave-one-out to match how the observed excess is centred. Independence across lags overstates the spread of the maximum (neighbours share their window), so it errs toward rejecting. Also: the positive branch now says to re-run before acting, because one run of this report has demonstrably not been a result; and MI_LAG_MAX_PROFILE caps the retained-draw matrix rather than trusting a derived m_historyBars. Read-only diagnostic. No input, topology or label change: no retrain, and a training run already in flight stays valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 16:28:57 -04:00
#define MI_LAG_MAX_PROFILE 32
//--- Draws per candidate in the geometry scan. Above the ranking-only 20 because these draws also build
//--- the FAMILY-WISE null, and a max-statistic lives in the upper tail where 20 draws are thinnest.
fix: gate the barrier-geometry winner on a family-wise null, not its own The scan ends by printing "set SL_Mode/TP_Mode to <winner> and retrain". That advisory fired on `bestExcess > cfgExcess * 1.5` - a ratio between two numbers, with no test that either is distinguishable from zero. bestExcess is a MAXIMUM over the eligible candidates. The maximum of several draws from a null sits well above any single draw from it, so a max-shaped statistic tested against a single-candidate null crowns a winner on noise almost every time. On SP500 H1 the winner is 2:8 at +0.00081 nats - and the lag profile committed in 3271f1e measures the pure-noise swing on this exact data at +/-0.0004, peaking at +0.00042 with nothing clearing its own null at any lag. The advisory was one ratio away from talking us into relabelling and retraining all four topologies to chase that. So build the null OF THE MAXIMUM: retain every candidate's permutation draws, take one draw from each candidate, keep the largest, repeat. The winner must beat that distribution. - draws centred LEAVE-ONE-OUT, so a draw is centred by a mean excluding it - exactly how the observed score is centred. Centring a draw by a mean that contains it shrinks it toward zero and would deflate the null. - only ELIGIBLE candidates enrol: the family the max was taken over is the family to correct for, and a clamped or sub-minRR pairing can never win. rrOK hoisted above the draws for this. - draws per candidate 20 -> MI_GEOMETRY_PERMUTATIONS (40): they now have to resolve an upper tail, which is where 20 draws are thinnest. - MI_GEOMETRY_ALPHA 0.05, stricter than the lag profile's: a wrong lookback costs input width, a wrong geometry costs a full retrain from era 0. Independence across candidates overstates the spread of the max (the real candidates share features and overlapping label windows), so the gate errs toward rejecting - the safe direction when passing costs a retrain. Read-only diagnostic. No input, topology or label change: no retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 14:04:16 -04:00
#define MI_GEOMETRY_PERMUTATIONS 40
//--- Family-wise significance for the geometry winner. Stricter than MI_LAG_ALPHA because acting on it
//--- means RELABELLING and retraining every topology from era 0.
fix: gate the barrier-geometry winner on a family-wise null, not its own The scan ends by printing "set SL_Mode/TP_Mode to <winner> and retrain". That advisory fired on `bestExcess > cfgExcess * 1.5` - a ratio between two numbers, with no test that either is distinguishable from zero. bestExcess is a MAXIMUM over the eligible candidates. The maximum of several draws from a null sits well above any single draw from it, so a max-shaped statistic tested against a single-candidate null crowns a winner on noise almost every time. On SP500 H1 the winner is 2:8 at +0.00081 nats - and the lag profile committed in 3271f1e measures the pure-noise swing on this exact data at +/-0.0004, peaking at +0.00042 with nothing clearing its own null at any lag. The advisory was one ratio away from talking us into relabelling and retraining all four topologies to chase that. So build the null OF THE MAXIMUM: retain every candidate's permutation draws, take one draw from each candidate, keep the largest, repeat. The winner must beat that distribution. - draws centred LEAVE-ONE-OUT, so a draw is centred by a mean excluding it - exactly how the observed score is centred. Centring a draw by a mean that contains it shrinks it toward zero and would deflate the null. - only ELIGIBLE candidates enrol: the family the max was taken over is the family to correct for, and a clamped or sub-minRR pairing can never win. rrOK hoisted above the draws for this. - draws per candidate 20 -> MI_GEOMETRY_PERMUTATIONS (40): they now have to resolve an upper tail, which is where 20 draws are thinnest. - MI_GEOMETRY_ALPHA 0.05, stricter than the lag profile's: a wrong lookback costs input width, a wrong geometry costs a full retrain from era 0. Independence across candidates overstates the spread of the max (the real candidates share features and overlapping label windows), so the gate errs toward rejecting - the safe direction when passing costs a retrain. Read-only diagnostic. No input, topology or label change: no retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 14:04:16 -04:00
#define MI_GEOMETRY_ALPHA 0.05
//--- Ceiling on scanned candidates, sizing the fixed draw matrix; the loop still skips ineligible pairs.
fix: gate the barrier-geometry winner on a family-wise null, not its own The scan ends by printing "set SL_Mode/TP_Mode to <winner> and retrain". That advisory fired on `bestExcess > cfgExcess * 1.5` - a ratio between two numbers, with no test that either is distinguishable from zero. bestExcess is a MAXIMUM over the eligible candidates. The maximum of several draws from a null sits well above any single draw from it, so a max-shaped statistic tested against a single-candidate null crowns a winner on noise almost every time. On SP500 H1 the winner is 2:8 at +0.00081 nats - and the lag profile committed in 3271f1e measures the pure-noise swing on this exact data at +/-0.0004, peaking at +0.00042 with nothing clearing its own null at any lag. The advisory was one ratio away from talking us into relabelling and retraining all four topologies to chase that. So build the null OF THE MAXIMUM: retain every candidate's permutation draws, take one draw from each candidate, keep the largest, repeat. The winner must beat that distribution. - draws centred LEAVE-ONE-OUT, so a draw is centred by a mean excluding it - exactly how the observed score is centred. Centring a draw by a mean that contains it shrinks it toward zero and would deflate the null. - only ELIGIBLE candidates enrol: the family the max was taken over is the family to correct for, and a clamped or sub-minRR pairing can never win. rrOK hoisted above the draws for this. - draws per candidate 20 -> MI_GEOMETRY_PERMUTATIONS (40): they now have to resolve an upper tail, which is where 20 draws are thinnest. - MI_GEOMETRY_ALPHA 0.05, stricter than the lag profile's: a wrong lookback costs input width, a wrong geometry costs a full retrain from era 0. Independence across candidates overstates the spread of the max (the real candidates share features and overlapping label windows), so the gate errs toward rejecting - the safe direction when passing costs a retrain. Read-only diagnostic. No input, topology or label change: no retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 14:04:16 -04:00
#define MI_GEOMETRY_MAX_CANDIDATES 12
//--- Standard errors a checkpoint's directional precision must clear chance by to be deployable.
fix(autotune): MI scorer read an array nobody filled; add the permutation floor THE TUNER WAS A SILENT NO-OP. Every chart logged auto-tune complete - 17 candidate settings scored in ~139s, feature/label mutual information 0.0000 -> 0.0000 nats (no improvement) 0.0000 is not a weak result, it is a broken measurement: finite-sample MI is biased UPWARD, so even pure noise scores above zero. Cause: ScoreCurrentParamsByMI called BufferTempDataCompute(), which APPENDS the bar's features to TempData and never touches m_featureCache - only the caching wrapper BufferTempData() writes that array. It then read m_featureCache, which ReInitADIndicators had just invalidated. Every column came back constant, FeatureColumnMI returned 0 for all of them, and all 17 candidates tied at exactly zero. 139 s per chart to return the settings it started with. Now reads the values back out of TempData, where they actually land. And an exactly-zero best score is called out as a fault rather than reported as "no improvement", because that is what it is. ADDED: a PERMUTATION BASELINE, which is the diagnostic this project has been missing. MI's finite-sample bias is ~(bins-1)(classes-1)/(2n) nats - at these sample sizes the same order as any real edge in this domain - so a raw MI figure is uninterpretable on its own. Shuffling the labels destroys every genuine association while leaving sample size, binning and class proportions intact, so the score it produces IS this dataset's noise floor, measured rather than approximated. The log now reads feature/label information - X nats against a shuffled-label floor of Y and says outright whether the features carry usable information about the target. It needs no training, no topology and no convergence, so unlike every accuracy number in this codebase it cannot be confounded by an optimizer or an objective. If the score sits on the floor, no change of architecture can help - which is the question the last three days of zero-edge results have been circling. DEPLOY FLOOR: `dirPrecPct > chancePrecPct` passed anything above chance by any amount. At ~11,000 directional calls the standard error of the precision estimate is ~0.4pp, so that gate was accepting sub-one-sigma noise - the perceptron deployed at edge +0pp on 2026-08-01. Now requires EDGE_MIN_SIGMAS (2.0) standard errors above chance, computed from the actual call count, so the bar scales with the evidence instead of needing a hand-picked constant. Recorded with it, because it is why chance is the right reference at all: under a driftless random walk P(touch +k*ATR before -m*ATR) = m/(m+k), and the break-even win rate for a k:m reward:risk trade is ALSO m/(m+k). The label's own base rate IS the break-even rate, at every SL/TP setting. So "beats chance" and "is profitable" are the same test, and no choice of SL/TP can manufacture an edge - only prediction can. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:05:50 -04:00
#define EDGE_MIN_SIGMAS 2.0
//--- Edge the geometry adoption must at least be ABLE to certify before switching. Deliberately generous
//--- - 10pp over break-even is more than anything measured here - because the guard catches pairings
//--- that are hopeless rather than merely hard.
fix(geometry): do not adopt a pairing that cannot be certified - information and detectability are different objectives Closes a gap 62a719f opened. Making ReportBarrierGeometryScan authoritative put two objectives in charge of one decision without reconciling them: - the scan maximises entry-time INFORMATION, in nats; - the deploy gate needs enough INDEPENDENT observations to certify an edge. They pull opposite ways. A wider pairing carries more information per call AND takes longer to resolve, and overlapping labels are worth ~1/L each - so tripling the horizon divides the independent sample by ~3 and multiplies the standard error the gate has to beat by ~sqrt(3). USDJPY's current winner is exactly that trade: 2:8 at h192 against an incumbent 1.61:3.21 at h64. Until today the adoption was inert so it never mattered; from 62a719f it decides the geometry, and it would have made that swap silently on the next fresh run. The guard inverts the deploy identity for the WINNER's own pairing - certifying an edge d needs z^2 p(1-p)/d^2 independent calls, at that pairing's break-even - and compares it against what the OOS window can physically supply at that horizon. If even a generous 10pp edge is out of reach, the pairing is not adopted and the log says it lost on detectability rather than on information. Conservative by construction: the horizon is an UPPER bound on the mean label lifespan, so oosBars/horizon is a LOWER bound on available independent observations. The guard therefore only fires when the pairing is hopeless, never merely hard. This is the same quantity ReportDetectability publishes per run, applied at the one moment it can still change a decision instead of after the geometry is already pinned. More information per trade is worth nothing if it buys too few independent trades to prove it - WIDTH is not free, and on this window it is the binding constraint, not the nats. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:47:37 -04:00
#define ADOPT_MIN_DETECTABLE_EDGE 0.10
//--- BOTH-DIRECTIONS FLOOR. The perceptron reported "Sell:0%" in all 41 of its eras, cleared on Buy
//--- alone at 36.6% vs 34%, and deployed.
fix: the Adam second moment was never Adam - all four tiers Root cause of the B=32 regression, and it predates F4 entirely. Every Adam kernel stored v already square-rooted and then fed that stored value back in as if it were the variance: v_new = sqrt(b2 * v_old + (1 - b2) * g^2) That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below unit scale, so the denominator stops tracking the gradient and Adam degrades into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll (batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where a scale-invariant optimizer gives the same distance for both. After the fix all six magnitudes read 1.199 and v tracks |g| exactly. It hit conv/LSTM specifically because they sit behind a batch-norm with running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep in the degraded regime - while the dense stack near the loss stayed in the working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/ 0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already squared v back for gamma/beta and its comment named the kernels as wrong, which is exactly why gamma/beta kept training while the stages behind froze. Persisted .nnw needs no migration - v keeps its std-dev meaning. Also, the two ways F4 exposed it, both mine: - No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods (Krizhevsky 2014; Granziol et al. 2022), applied once in InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD. - Plateau patience denominated in eras, so raising B made the ladder 32x more impatient in its only unit. PAI converged at era 41 on ~49k updates where the same config had been finding new bests at era 1028. TrainPlateauPatienceEras() stretches it by the same sqrt(B). TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23 eras per stage, not 8 -> 45). Both helpers are identities at B=1. Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into the ranking key rather than checked at deploy time so a one-sided era cannot become best-so-far in the first place. Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded CAppDialog teardown that sat ahead of it - the same ordering inversion the rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit (vs ~1.1 s for the three that finished) having reached none of its cleanup, so its arrows stayed on the chart. Steps are now timed in the log. PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now walks every object type and reports the object counts when both are zero. Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt. FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
#define DEPLOY_MIN_SIDE_RECALL_PCT 10.0
//--- FAMILY-WISE DEPLOYMENT GATE. EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the
//--- MAXIMUM over every era - the one construction this project has repeatedly proven crowns noise.
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
#define DEPLOY_FAMILY_WISE_ALPHA 0.05
//--- No oversampling constants: data-level class-balance oversampling went on 2026-07-31, and the
//--- imbalance is corrected analytically in the gradient by the logit-adjusted loss. TRIPLE-BARRIER
//--- LABELS (Lopez de Prado, ch. 3).
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
#define BARRIER_TIE_GOES_TO_STOP 1
//--- Vertical (time) barrier, in bars. NOT in the weights fingerprint: a filename keyed on a
//--- measured quantity orphans a trained model the moment the measurement moves.
fix(labels): the 128-bar horizon ceiling was truncating the shipped label The corrected geometry scan exposed something bigger than the geometry question it was asked. Every pairing from 2:6 upward came back CLAMPED - including 2:6, the SHIPPED configuration. First-passage time for a driftless walk leaving [-m,+k] goes as m*k, and the measured swing median here is ~12 bars at m*k=1, so 2:6 wants ~144 bars and 3:10 wants ~360. The ladder stopped at 128. A clamped label stops meaning "does the target come before the stop" and quietly becomes "...within 128 bars", while the deployed EA holds until SL or TP with no bar limit. So the target the models have been trained on all along was not the strategy the EA executes, and the trades it silently reclassified as Neutral were the SLOW WINNERS - precisely the ones a 1:3 barrier exists to capture. Timeout share stayed ~0% throughout, which is why this never showed up: the truncation lands in Neutral, not in the timeout counter that was watching for it. Ladder extended to 384 (12..128, 192, 256, 384) so every selectable geometry gets an honest horizon. Cost is one embargo of at most 384 bars out of ~38k. Second fix, same class of error as the H(Y) one: the scan's "best eligible" was 2:2, a 1:1 barrier, against a shipped Min_Risk_Reward_Ratio of 1:2. Training four topologies on that target would have produced a model whose every setup is rejected at the door - the exact failure behind four consecutive Market rejections for "no trading operations". Sub-minRR geometries are now ineligible and marked [<minRR], printed rather than hidden. Also drops the dense-depth tag from the display name ("Perceptron 3L" -> "Perceptron"). Depth is derived, so it names nothing a user chose; the config tag [PAI-0be2] already disambiguates concurrent charts and does it for every input rather than one. Full topology still logged by "config -". Compiles 0 errors / 0 warnings, standard and Market. Build tag horizon-384-v1. Changes the LABEL for every geometry, so the next scan supersedes the previous numbers - and a retrain is required before any model trained under the truncated target means anything. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:54:45 -04:00
#define BARRIER_HORIZON_LADDER_COUNT 11
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
#define BARRIER_HORIZON_MIN 12
fix(labels): the 128-bar horizon ceiling was truncating the shipped label The corrected geometry scan exposed something bigger than the geometry question it was asked. Every pairing from 2:6 upward came back CLAMPED - including 2:6, the SHIPPED configuration. First-passage time for a driftless walk leaving [-m,+k] goes as m*k, and the measured swing median here is ~12 bars at m*k=1, so 2:6 wants ~144 bars and 3:10 wants ~360. The ladder stopped at 128. A clamped label stops meaning "does the target come before the stop" and quietly becomes "...within 128 bars", while the deployed EA holds until SL or TP with no bar limit. So the target the models have been trained on all along was not the strategy the EA executes, and the trades it silently reclassified as Neutral were the SLOW WINNERS - precisely the ones a 1:3 barrier exists to capture. Timeout share stayed ~0% throughout, which is why this never showed up: the truncation lands in Neutral, not in the timeout counter that was watching for it. Ladder extended to 384 (12..128, 192, 256, 384) so every selectable geometry gets an honest horizon. Cost is one embargo of at most 384 bars out of ~38k. Second fix, same class of error as the H(Y) one: the scan's "best eligible" was 2:2, a 1:1 barrier, against a shipped Min_Risk_Reward_Ratio of 1:2. Training four topologies on that target would have produced a model whose every setup is rejected at the door - the exact failure behind four consecutive Market rejections for "no trading operations". Sub-minRR geometries are now ineligible and marked [<minRR], printed rather than hidden. Also drops the dense-depth tag from the display name ("Perceptron 3L" -> "Perceptron"). Depth is derived, so it names nothing a user chose; the config tag [PAI-0be2] already disambiguates concurrent charts and does it for every input rather than one. Full topology still logged by "config -". Compiles 0 errors / 0 warnings, standard and Market. Build tag horizon-384-v1. Changes the LABEL for every geometry, so the next scan supersedes the previous numbers - and a retrain is required before any model trained under the truncated target means anything. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:54:45 -04:00
#define BARRIER_HORIZON_MAX 384
//--- Fallback when the ZigZag scan finds too few pivots for a median. Mid-ladder, and it logs when used.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
#define BARRIER_HORIZON_FALLBACK 32
//--- Confirmed pivots required before the median is trusted rather than the fallback.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
#define BARRIER_HORIZON_MIN_SAMPLES 20
//--- Share of bars one class must hold before the era-0 output-bias seed fires. A +-3.0 bias seed is a
//--- correction at a 94%-Neutral prior and a distortion at a 50% one.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
#define COLD_START_SEED_MIN_DOMINANCE 0.70
//--- Reduce-on-regression learning-rate decay. g_eta is read fresh by every weight-update call on
//--- every backend, so shrinking it takes effect on the next backProp() everywhere at once.
#define ETA_DECAY_REGRESSION_PCT 5.0 // only decay after a real regression, not per-era noise
#define ETA_DECAY_FACTOR 0.7
//--- 1e-5, not 1e-4: against the 3e-4 ceiling the old floor left the schedule a 3x dynamic range, so
//--- three decays pinned it and "reduce LR on regression" could never settle an oscillating run. The
//--- recovery bump still climbs back at 1/0.7 per new best.
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
#define ETA_MIN 0.00001
//--- ComputeFirstLayerWidth() constants. SECONDS_PER_YEAR is the mean Julian year, MQL5's own
//--- convention. MARKET_OPEN_FRACTION allows for closed hours and weekends - anything in 0.6-0.85
//--- lands on the same ladder rung.
refactor(ai): derive the first dense layer's width instead of asking for it InitialNeurons was an input whose only defensible value depends on two things the user cannot see when picking from a dropdown: how wide the input vector ended up after feature selection, and how much in-sample data the study period actually yields. Left to a hand-picked constant it was badly wrong - 500 units against a 420-wide input is 210,500 weights, 72% of a 292,583-weight model, against ~36,500 training bars of which only ~2,236 are directional. That is 6.6 weights per training bar, and it EXPANDS a set of highly correlated inputs rather than compressing them. The symptom was already in the logs and had been read as a depth problem: the shallowest topology consistently beat the deepest (perceptron 52.7% balanced, hybrid 41.3%). Over-parameterization predicts that ordering just as well as covariate shift does, and only one of the two had been addressed. ComputeFirstLayerWidth() budgets roughly one first-layer weight per in-sample bar. Measured across the configurations in use: M15 10y -> 256 units, 129,071 weights, 0.73 per bar H1 10y -> 64 units, 28,727 weights, 0.65 per bar H4 10y -> 16 units, 7,559 weights, 0.68 per bar Two design points that matter: - It estimates in-sample bars from the STUDY PERIOD and timeframe, not from Bars(). What is downloaded grows over a terminal's lifetime, and a topology that widened as history filled in would re-key its own weights file and discard a trained model. - The result is snapped down to a coarse power-of-two ladder, so the estimate would have to be wrong by ~2x to change the answer. Every field it reads is already part of the weights-filename fingerprint, so the derived value needs no fingerprint entry of its own. The public setter is removed - it could only have been called after construction, and would either be ignored or silently re-key the model mid-run. Where the data cannot support even the floor (D1 over 10 years is under 2,000 bars) it now says so and names the fixes, rather than quietly training a model with more weights than examples. The DB config fingerprint drops the term too, which re-keys existing pattern databases once - correct, since a model an order of magnitude smaller should not inherit the old one's win-rate history. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 13:01:16 -04:00
#define SECONDS_PER_YEAR 31557600.0
#define MARKET_OPEN_FRACTION 0.72
//--- Ceiling on how much of the head's LOGIT RANGE the logit-adjustment offsets may consume. Menon
//--- et al. assume an UNBOUNDED head.
fix(ai): cap logit-adjustment strength to the head's usable logit range tau=1.0 inverted the collapse instead of curing it. The head is SIGMOID, so each output is bounded to [0,1] and the widest logit gap the net can express between two classes is CLASS_LOGIT_SCALE * (1-0) = 6. The offsets are tau*log(prior_c), whose spread on this 30:1 imbalance is 3.42 - so tau=1.0 spent 57% of the ENTIRE expressible range on the prior correction. The network did the only thing available to it: saturate Buy/Sell outputs to 1.0 to overcome a -3.42 training handicap. The offsets are absent at inference, so that surplus made every bar directional. Measured across all five still-training charts: Neutral recall 0%, directional calls on ~100% of bars, win rate 5-7% against a ~6% base rate - no information whatsoever - while balanced accuracy read a flattering 58-64% because two of its three terms sat near 95%. OOS accuracy 6%. Menon et al. assume an unbounded logit head where a 3.42 shift is negligible against the reachable range. It is not negligible here, so the strength is now expressed RELATIVE to the range actually available: tau_eff = min(tau_cfg, LOGIT_ADJUST_MAX_RANGE_FRACTION * SCALE / spread) At 20% that gives tau 0.35 on this data. Deliberately a fraction rather than a tau ceiling: it stays correct if CLASS_LOGIT_SCALE changes, if the head becomes unbounded, or on any symbol whose imbalance differs. The input remains effective below the cap, so dialling it down needs no rebuild. Simulated at a signal strength where the task is genuinely learnable, the precision/recall frontier is monotone: tau 1.0 -> 49.6% call rate at 6.4% precision (base rate 6.1%, i.e. worthless); tau 0.35 -> 2.0% at 15.5%; tau 0.15 -> 0.2% at 33.3%. The capped value lands in the same regime the pre-logit-adjustment run occupied (1-6% of bars at 20-35% win rate). Also logs the measured priors, the spread, and whether the cap bound. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 23:20:07 -04:00
#define LOGIT_ADJUST_MAX_RANGE_FRACTION 0.20
//--- Minimum directional call rate for deployability, as a FRACTION OF THE TRUE DIRECTIONAL BASE RATE
//--- rather than an absolute percentage - a model calling a direction a quarter as often as one occurs
//--- is sparse but usable; one calling ten times a decade is not, however precise those ten were.
feat(ai): rank checkpoints on directional precision, not balanced accuracy Balanced accuracy is maximized by exactly the model this system must never deploy. Measured frontier at fixed signal strength, base rate 6.1%: tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0% tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3% tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5% It rises monotonically as the model calls MORE and is right LESS, because two of its three terms are directional recalls that a call-everything model drives to ~95%, while the Neutral term it sacrifices counts for only a third. The 2026-07-29 run landed exactly there: balanced 58-64% while calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base rate. Only the per-class recall floor stopped those deploying - a guard doing the job the objective should have been doing - and that same guard also rejected the genuinely useful sparse-but-precise checkpoints. Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how many were right. That is what a trading edge is. Two anti-degenerate floors bracket it, since precision alone is trivially maximized by calling almost nothing: coverage must reach a fraction of the true directional base rate (derived, not configured - it adapts to any symbol/timeframe/label rule), and precision must at least beat that base rate. Against the same frontier the deploy order inverts from tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first) to tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage) Balanced accuracy is kept in the log as a diagnostic and marked as such, so a run where the two disagree - the signature of an over-caller - is visible at a glance. MinRecall no longer decides what ships; it now only drives the diagnostic recall line and is a candidate for removal. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
#define MIN_COVERAGE_FRACTION_OF_BASE_RATE 0.25
//--- DIRECTIONAL CONFIDENCE THRESHOLD. The decision RULE carries the trading policy rather than
//--- distorting the loss (Elkan 2001). Fatal HERE because FitDirConfThreshold branches on the SIGN
//--- of (p - break-even), and a memorized curve never shows p < p0, so the get-more-selective
//--- branch could never fire.
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
#define DIR_CONF_THRESHOLD_BINS 50
//--- Below this the histogram is too sparse to pick an operating point from. The model then KEEPS THE
//--- PREVIOUS ERA'S THRESHOLD rather than falling back to 0.0 - "trade every bar" is the most dangerous
//--- setting in the range and must never be what a failed measurement decays to.
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
#define DIR_CONF_MIN_FIT_CALLS 200
//--- Share of the IS span held out to fit the operating point. 15% of ~38k bars is ~5.7k, ~28x the
//--- minimum, so the fit is never sparse. Larger buys precision at a direct cost in training data.
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
#define DIR_CONF_CALIB_PCT_OF_IS 15
//--- EXCURSION-SIZE HEAD (see AIBase\Excursion.mqh). Hidden width is deliberately small: the question
//--- has a known low-dimensional answer (volatility clustering), and era time here is taken from a
//--- classifier already at ~300 s/era.
feat: excursion-size head (Stage 1, measurement only) Direction is closed - normalised asymmetry fails on three instruments with a working positive control, and the classifier's own best-of-999 era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is a different question and RANGE clears at ~4x its null. Checked the denomination before building on that, since the source memo warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is predictable" is a claim about travel RELATIVE to current ATR, not a restatement of "ATR is autocorrelated". It is exactly the part a fixed multiple (stop 3.31*ATR, target 1.64*ATR) discards. A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches ladder rung k) upward and downward. Survival parameterisation rather than regressing the multiple, because it needs nothing new from CNet: sigmoid outputs and the per-neuron delta the `total != 3` branch already applies (a quantile head would need a linear activation and a pinball gradient in Network.mqh, Network.cl and the DirectML path, on a class four topologies share). Targets are free - m_ladderUpAt already records first-touch age per rung with 0 meaning never reached. Separate net, not extra outputs on the classifier: more outputs would change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push the count off 3 - the exact condition backProp uses to select the joint softmax gradient the 3-class head depends on. The classifier is bit-for-bit unaffected and this is removable without trace. STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the constant per-rung base rate - the baseline a fixed ATR multiple already assumes - with both predictors fitted IS and evaluated OOS, so neither gets a look at the test set. Positive skill justifies Stage 2 (drive SL/TP and sizing off ExcursionQuantile, which is defined and deliberately uncalled). Zero or negative means ATR already carries everything and Stage 2 must not be built. Trains only on primary occurrences: the replay queue oversamples for CLASS balance, and a direction-balanced sample is a biased SIZE sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
#define EXCURSION_HIDDEN_UNITS 24
//--- Below this many held-out bars the Brier skill score is noise and no verdict is printed.
#define EXCURSION_MIN_SCORED 500
//--- Skill the head must beat before Stage 2 is justified. Not zero: replacing a constant that cannot
//--- fail with a learned quantity that can must buy more than the era-to-era wobble of the estimate.
feat: excursion-size head (Stage 1, measurement only) Direction is closed - normalised asymmetry fails on three instruments with a working positive control, and the classifier's own best-of-999 era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is a different question and RANGE clears at ~4x its null. Checked the denomination before building on that, since the source memo warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is predictable" is a claim about travel RELATIVE to current ATR, not a restatement of "ATR is autocorrelated". It is exactly the part a fixed multiple (stop 3.31*ATR, target 1.64*ATR) discards. A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches ladder rung k) upward and downward. Survival parameterisation rather than regressing the multiple, because it needs nothing new from CNet: sigmoid outputs and the per-neuron delta the `total != 3` branch already applies (a quantile head would need a linear activation and a pinball gradient in Network.mqh, Network.cl and the DirectML path, on a class four topologies share). Targets are free - m_ladderUpAt already records first-touch age per rung with 0 meaning never reached. Separate net, not extra outputs on the classifier: more outputs would change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push the count off 3 - the exact condition backProp uses to select the joint softmax gradient the 3-class head depends on. The classifier is bit-for-bit unaffected and this is removable without trace. STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the constant per-rung base rate - the baseline a fixed ATR multiple already assumes - with both predictors fitted IS and evaluated OOS, so neither gets a look at the test set. Positive skill justifies Stage 2 (drive SL/TP and sizing off ExcursionQuantile, which is defined and deliberately uncalled). Zero or negative means ATR already carries everything and Stage 2 must not be built. Trains only on primary occurrences: the replay queue oversamples for CLASS balance, and a direction-balanced sample is a biased SIZE sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
#define EXCURSION_SKILL_USEFUL_PCT 2.0
//--- Minimum DISJOINT (non-overlapping-horizon) observations before the tally decides anything.
//--- This used to be 200, sized as "~16k scored bars over a 64-bar horizon leaves ~250 independent
//--- ones".
fix(excursion): the sample gate asked for more windows than the configuration can contain EXCURSION_MIN_DISJOINT was 200, sized as "~16k scored bars over a 64-bar horizon leaves ~250 independent ones". The head scores the OOS SLICE, not the history. At a 30% split and a 32-bar horizon the ceiling is 4691/32 = 147, so 200 was unreachable and every era printed "[disjoint sample too small]", which an operator reads as "wait longer". Unpassable by construction - the identical failure this file already documents one gate down, one layer up. The trail gate inherited it: m_excTrailScored is a subset of the disjoint bars, so it failed the same 200 for the same reason, at 130. Raising OOSSplit or shortening the horizon would clear it and would be fitting the experiment to the answer. Instead, ask the question the count was standing in for - is the skill bigger than its own noise: - The scorer banks one paired Brier difference per DISJOINT window over the decision rungs (base-head, and trail-head). Disjoint by construction, so no EffectiveSampleSize deflation applies - striding by the horizon is what buys that - and paired on identical bars, so the correlation between the two predictors cancels instead of needing to be estimated. - passDj and passTrail now require skill >= EXCURSION_SKILL_USEFUL_PCT AND >= 2 sigma, with the count reduced to a sanity floor of 30. - Both sigmas print on the verdict line. This is not a lowered bar. The 2% skill requirement, the oracle control and the monotone test are untouched, and the sigma test can fail where the count test never spoke: if +8.5% is noise across 147 windows, it will now say so. DecisionRungMask() is the single definition of "rung the decision depends on", called by both the scorer and the report, so the standard error is computed over exactly the rungs the skill score is. The report's inline copy of the bracketing test is gone. Also prints whether the disjoint count is BELOW ITS CEILING or at it, so "not enough yet" and "not in this configuration" stop reading the same. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 23:06:49 -04:00
#define EXCURSION_MIN_DISJOINT_SANITY 30
//--- Sigmas the paired per-window Brier difference must clear. Two-sided 2 sigma, the same bar every
//--- other decision in this codebase is held to. Measured on DISJOINT windows, so no EffectiveSampleSize
//--- deflation applies - striding by the horizon is precisely what buys that.
#define EXCURSION_MIN_SIGMA 2.0
fix: the excursion gate would have passed Stage 2 on an artifact I made Second-opinion review killed the +4.2% far-rung result, correctly, and the mechanism is my own bug. A head trained toward {0.05,0.9} converges to 0.05+0.85p, so its bias is 0.05-0.15p: negative where p is near 1, POSITIVE where p < 1/3, growing monotonically as the rung gets farther. Against a baseline frozen at the IS rate, an upward-biased head scores positive Brier skill whenever the OOS rate merely sits above the IS rate. Predicted signature: huge negatives near, ~zero at p=1/3, growing positives far. Observed: -82% ... -0.6% ... +1.2/+2.7/+4.2. The far rungs were not the clean end of a distorted measurement, they were the other face of the same artifact. Everything before 25aca83 is void. The gate was a bare `skill >= 2%` point estimate over 8 rungs x 4 topologies x N eras, reported per era - a best-of-~300 with no interval and no multiplicity control, which is the shape of the four traps already documented here. It now needs FOUR things at once: DECISION RUNGS only the rungs ExcursionQuantile actually reads at the live geometry (target 1.62, stop 3.31 ATR), fixed before looking. Skill at 5 ATR is skill about a distance no order is placed at - and the TARGET side currently interpolates 1.5/2.0, which measured -2.2% and -1.3%. DISJOINT SAMPLE one bar per horizon. Adjacent bars share 63 of 64 horizon bars, so ~16k scored bars is ~250 independent ones and every SE over the full set is ~8x understated. VS ORACLE the best constant achievable ON THE SCORED BLOCK, closed form from H and n (Brier = H*(1-H/n)). A head that learned only a LEVEL nearer the OOS rate than the frozen IS constant scores positive against the old baseline and <= 0 here. This is the control that separates per-bar skill from base-rate drift. MONOTONE CURVE P(reach k) must be non-increasing in k. Nothing constrained 8 independent sigmoids to obey that, and ExcursionQuantile returns the FIRST crossing - so a tangled curve is misread exactly where the head is least sure. Counted and reported, not silently used. The pass message now also states what a pass would and would not buy: expectancy is -costs at zero directional edge whatever the stop distance, and under prop DD limits LOWER variance also lowers P(reach target before limit), so "better drawdown" is a choice of failure mode, not a win. Still owed before any Stage 2: a race against a trailing-quantile incumbent and a vol-feature logistic. Beating a frozen global constant is the weakest admissible bar for replacing a global constant. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:49:32 -04:00
//--- Share of bars whose predicted survival curve is non-monotone. Above this the 8 sigmoids are not
//--- describing one distribution and ExcursionQuantile's first-crossing read is undefined.
fix: the excursion gate would have passed Stage 2 on an artifact I made Second-opinion review killed the +4.2% far-rung result, correctly, and the mechanism is my own bug. A head trained toward {0.05,0.9} converges to 0.05+0.85p, so its bias is 0.05-0.15p: negative where p is near 1, POSITIVE where p < 1/3, growing monotonically as the rung gets farther. Against a baseline frozen at the IS rate, an upward-biased head scores positive Brier skill whenever the OOS rate merely sits above the IS rate. Predicted signature: huge negatives near, ~zero at p=1/3, growing positives far. Observed: -82% ... -0.6% ... +1.2/+2.7/+4.2. The far rungs were not the clean end of a distorted measurement, they were the other face of the same artifact. Everything before 25aca83 is void. The gate was a bare `skill >= 2%` point estimate over 8 rungs x 4 topologies x N eras, reported per era - a best-of-~300 with no interval and no multiplicity control, which is the shape of the four traps already documented here. It now needs FOUR things at once: DECISION RUNGS only the rungs ExcursionQuantile actually reads at the live geometry (target 1.62, stop 3.31 ATR), fixed before looking. Skill at 5 ATR is skill about a distance no order is placed at - and the TARGET side currently interpolates 1.5/2.0, which measured -2.2% and -1.3%. DISJOINT SAMPLE one bar per horizon. Adjacent bars share 63 of 64 horizon bars, so ~16k scored bars is ~250 independent ones and every SE over the full set is ~8x understated. VS ORACLE the best constant achievable ON THE SCORED BLOCK, closed form from H and n (Brier = H*(1-H/n)). A head that learned only a LEVEL nearer the OOS rate than the frozen IS constant scores positive against the old baseline and <= 0 here. This is the control that separates per-bar skill from base-rate drift. MONOTONE CURVE P(reach k) must be non-increasing in k. Nothing constrained 8 independent sigmoids to obey that, and ExcursionQuantile returns the FIRST crossing - so a tangled curve is misread exactly where the head is least sure. Counted and reported, not silently used. The pass message now also states what a pass would and would not buy: expectancy is -costs at zero directional edge whatever the stop distance, and under prop DD limits LOWER variance also lowers P(reach target before limit), so "better drawdown" is a choice of failure mode, not a win. Still owed before any Stage 2: a race against a trailing-quantile incumbent and a vol-feature logistic. Beating a frozen global constant is the weakest admissible bar for replacing a global constant. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:49:32 -04:00
#define EXCURSION_MAX_MONO_VIOL_PCT 5.0
//--- TRAILING-CLIMATOLOGY window over RESOLVED outcomes only - the real incumbent for "replace a global
//--- ATR multiple", since a rolling rung frequency adapts to the regime and needs no model at all. The
//--- head's margin over THIS, not over a frozen constant, is what would justify 760 inputs.
feat: race the excursion head against a trailing-quantile incumbent Beating a frozen global constant is the weakest admissible bar for replacing a global constant. The honest incumbent is a rolling rung frequency: it adapts to the volatility regime - exactly what the head claims to predict - and needs no model, no 760 inputs and no training. Implemented as a ring of per-bar outcome bitmasks (32 rungs fit one ulong), sized horizon + EXCURSION_TRAIL_WINDOW. The newest `horizon` entries are held back UNRESOLVED: a bar's rung outcomes are only known one horizon later, so using them would be lookahead and would flatter the incumbent into an opponent the head could never fairly beat. Pass 3 walks oldest-to-newest, so "pushed more than horizon bars ago" is exactly "resolved by now". Each push is O(rungs), not O(window). The head's decision-rung Brier is pro-rated to the trailing estimate's coverage before the ratio, since the incumbent only scores bars where its window is warm. This line is worth reading on its own, independently of the head: if the trailing quantile beats the global constant, that is a cheap risk-control win available with no machine learning at all - and it is the same number either way, so the run answers both questions in one pass. The ring is deliberately NOT reset per era - it estimates the market, not the era, and re-warming 500 bars every era would leave the incumbent unusable over the first chunk of every scoring pass, handing the head a free win on exactly those bars. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:57:11 -04:00
#define EXCURSION_TRAIL_WINDOW 2000
//--- Resolved bars the trailing window must hold before its estimate may score anything.
feat: race the excursion head against a trailing-quantile incumbent Beating a frozen global constant is the weakest admissible bar for replacing a global constant. The honest incumbent is a rolling rung frequency: it adapts to the volatility regime - exactly what the head claims to predict - and needs no model, no 760 inputs and no training. Implemented as a ring of per-bar outcome bitmasks (32 rungs fit one ulong), sized horizon + EXCURSION_TRAIL_WINDOW. The newest `horizon` entries are held back UNRESOLVED: a bar's rung outcomes are only known one horizon later, so using them would be lookahead and would flatter the incumbent into an opponent the head could never fairly beat. Pass 3 walks oldest-to-newest, so "pushed more than horizon bars ago" is exactly "resolved by now". Each push is O(rungs), not O(window). The head's decision-rung Brier is pro-rated to the trailing estimate's coverage before the ratio, since the incumbent only scores bars where its window is warm. This line is worth reading on its own, independently of the head: if the trailing quantile beats the global constant, that is a cheap risk-control win available with no machine learning at all - and it is the same number either way, so the run answers both questions in one pass. The ring is deliberately NOT reset per era - it estimates the market, not the era, and re-warming 500 bars every era would leave the incumbent unusable over the first chunk of every scoring pass, handing the head a free win on exactly those bars. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:57:11 -04:00
#define EXCURSION_TRAIL_MIN_N 500
//--- Train the head on one primary bar in this many: excursion targets are strongly autocorrelated, so
//--- consecutive samples are near-duplicates, and this net's cost is per-dispatch rather than per-FLOP.
perf: the excursion head cost 3.6x era time - cut its dispatches ~250x Measured on exc-race-v3: LSTM era 300s -> 1087s (net 272->748s, "other" 30->337s). My estimate had been "single-digit percent". The cost is per-DISPATCH, not per-FLOP, and therefore hits EVERY backend: the head is 19k weights and ~2.4 GFLOP an era - seconds of arithmetic - but ~48k forward/backward calls x several layer submits each, and its 760-wide layer exceeds the CPU DLL's inline threshold so each one pays a real handoff. The classifier's own net time tripled too, from contention with a second pool on an already-full box. Three changes, all backend-neutral because they remove submits rather than tune threads: SCORE ONLY DISJOINT WINDOWS (~64x). Adjacent bars share all but one bar of their horizon, so 16k consecutive bars were always ~250 independent observations - the full-sample tally was never worth more than the disjoint one, it just quoted an n that was ~64x too large. Dropping it costs nothing statistically and removes 63 of every 64 forward passes. The two parallel tallies collapse into one, which is also less code. The trailing ring still advances on every bar: it needs the outcome SEQUENCE, and that is array lookups, not a forward pass. TRAIN ON EVERY 4th PRIMARY BAR (4x). The target is low-dimensional and strongly autocorrelated - neighbouring bars carry near-identical excursion information - so per-bar training buys resolution the target does not have. Strided on ATTEMPTS, not acceptances, so a stretch of unlabelled bars cannot silently change the spacing. OWN TIMING COLUMN. The head's passes were landing in the era line's "other" bucket, which is how a 3.6x regression read as an unexplained jump in the one column nobody attributes. A cost that cannot be seen in the timing line cannot be traded off against anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:38:29 -04:00
#define EXCURSION_TRAIN_STRIDE 4
refactor(ai): derive the first dense layer's width instead of asking for it InitialNeurons was an input whose only defensible value depends on two things the user cannot see when picking from a dropdown: how wide the input vector ended up after feature selection, and how much in-sample data the study period actually yields. Left to a hand-picked constant it was badly wrong - 500 units against a 420-wide input is 210,500 weights, 72% of a 292,583-weight model, against ~36,500 training bars of which only ~2,236 are directional. That is 6.6 weights per training bar, and it EXPANDS a set of highly correlated inputs rather than compressing them. The symptom was already in the logs and had been read as a depth problem: the shallowest topology consistently beat the deepest (perceptron 52.7% balanced, hybrid 41.3%). Over-parameterization predicts that ordering just as well as covariate shift does, and only one of the two had been addressed. ComputeFirstLayerWidth() budgets roughly one first-layer weight per in-sample bar. Measured across the configurations in use: M15 10y -> 256 units, 129,071 weights, 0.73 per bar H1 10y -> 64 units, 28,727 weights, 0.65 per bar H4 10y -> 16 units, 7,559 weights, 0.68 per bar Two design points that matter: - It estimates in-sample bars from the STUDY PERIOD and timeframe, not from Bars(). What is downloaded grows over a terminal's lifetime, and a topology that widened as history filled in would re-key its own weights file and discard a trained model. - The result is snapped down to a coarse power-of-two ladder, so the estimate would have to be wrong by ~2x to change the answer. Every field it reads is already part of the weights-filename fingerprint, so the derived value needs no fingerprint entry of its own. The public setter is removed - it could only have been called after construction, and would either be ignored or silently re-key the model mid-run. Where the data cannot support even the floor (D1 over 10 years is under 2,000 bars) it now says so and names the fixes, rather than quietly training a model with more weights than examples. The DB config fingerprint drops the term too, which re-keys existing pattern databases once - correct, since a model an order of magnitude smaller should not inherit the old one's win-rate history. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 13:01:16 -04:00
#define FIRST_LAYER_MIN_WIDTH 16
//--- Conv receptive field in BARS, a structural property of the front-end rather than an input: 3
//--- is the smallest window that can express a turning point (before/at/after), which is what
//--- ZigZag marks.
feat(ai): true multi-bar conv and true sequence LSTM CONV and LSTM were each configured as a strictly lossier perceptron, which is exactly what the panel showed: PAI 24% > CONV 18% > HYBRID 12% ~ LSTM 12%, monotone in how much reaches the dense stack (420 / 160 / 32 / 16). CONV - receptive field 1 -> 3 bars, and the pool is gone. Reading the reference kernels settled why 34d6aa4 killed CONV. FeedForwardConv emits POSITION-MAJOR output (matrix_o[out + window_out*i]), and FeedForwardProof is a flat contiguous max over `window` at stride `step`. On that layout any window <= window_out maxes ACROSS FILTERS within one position - it cannot pool over time at all. Our stage used window = step = filterCount: one max over all 8 filters per position, discarding 87.5% of the conv output and leaving only the argmax filter with gradient. That is a property of the reference's layout, not a porting bug, so there is no correct pool to swap in. Springenberg et al. ICLR 2015 is the answer already cited in this file: no pooling, get the hierarchy from strided convolution. The second conv went with it - its window was counted in raw elements while its comment claimed positions, so a "2-position" window actually spanned 2 filters of position 0. Filter count now derives from the WINDOW (RF * features / 2) instead of one bar, which at RF 3 was under-sizing the stage 3x. Shape: 20 bars x 21 -> 18 positions x 16 filters = 288. LSTM - sequence mode back on, forget bias 2.0 -> 1.0. The forward path rules out the "no gradient" reading of the 2026-07-30 failure: CPU_LSTMSeqForward starts every sample at h_0 = c_0 = 0 and unrolls that sample's own window, so nothing leaks between shuffled samples. Flat IS error + Neutral:100% is equally the signature of an output that does not vary with the input, and that is what bias 2.0 produces: c* = i*g/(1-sigmoid(b)) ~ 8.3*i*g, |c*| ~ 4.2, tanh pinned at 0.9995 with derivative 1e-3, so h_T is near-binary and set by the gate biases rather than the bars. Choosing 2.0 off the reach sweep was a method error - reach trades against saturation and the sweep never measured saturation. 1.0 is the Gers/Jozefowicz/Keras default and leaves tanh derivative ~0.1. Both builds 0/0. DLL unchanged (CPU_LSTMSeqForward/Backward already exported). Forces a retrain of CONV, LSTM and HYBRID - the .nnw pins architecture. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 17:42:33 -04:00
#define CONV_RECEPTIVE_FIELD_BARS 3
//--- Sequence-LSTM front-end. 1 = a real recurrence over bars with BPTT, gradient-checked to
//--- 2.3e-10 (DirectML\lstm_seq_gradcheck.cpp). 0 = the pre-2026-07-30 single gate step over the
//--- flattened input.
feat(ai): true multi-bar conv and true sequence LSTM CONV and LSTM were each configured as a strictly lossier perceptron, which is exactly what the panel showed: PAI 24% > CONV 18% > HYBRID 12% ~ LSTM 12%, monotone in how much reaches the dense stack (420 / 160 / 32 / 16). CONV - receptive field 1 -> 3 bars, and the pool is gone. Reading the reference kernels settled why 34d6aa4 killed CONV. FeedForwardConv emits POSITION-MAJOR output (matrix_o[out + window_out*i]), and FeedForwardProof is a flat contiguous max over `window` at stride `step`. On that layout any window <= window_out maxes ACROSS FILTERS within one position - it cannot pool over time at all. Our stage used window = step = filterCount: one max over all 8 filters per position, discarding 87.5% of the conv output and leaving only the argmax filter with gradient. That is a property of the reference's layout, not a porting bug, so there is no correct pool to swap in. Springenberg et al. ICLR 2015 is the answer already cited in this file: no pooling, get the hierarchy from strided convolution. The second conv went with it - its window was counted in raw elements while its comment claimed positions, so a "2-position" window actually spanned 2 filters of position 0. Filter count now derives from the WINDOW (RF * features / 2) instead of one bar, which at RF 3 was under-sizing the stage 3x. Shape: 20 bars x 21 -> 18 positions x 16 filters = 288. LSTM - sequence mode back on, forget bias 2.0 -> 1.0. The forward path rules out the "no gradient" reading of the 2026-07-30 failure: CPU_LSTMSeqForward starts every sample at h_0 = c_0 = 0 and unrolls that sample's own window, so nothing leaks between shuffled samples. Flat IS error + Neutral:100% is equally the signature of an output that does not vary with the input, and that is what bias 2.0 produces: c* = i*g/(1-sigmoid(b)) ~ 8.3*i*g, |c*| ~ 4.2, tanh pinned at 0.9995 with derivative 1e-3, so h_T is near-binary and set by the gate biases rather than the bars. Choosing 2.0 off the reach sweep was a method error - reach trades against saturation and the sweep never measured saturation. 1.0 is the Gers/Jozefowicz/Keras default and leaves tanh derivative ~0.1. Both builds 0/0. DLL unchanged (CPU_LSTMSeqForward/Backward already exported). Forces a retrain of CONV, LSTM and HYBRID - the .nnw pins architecture. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 17:42:33 -04:00
#define LSTM_SEQUENCE_MODE 1
//--- Where the dense taper ENDS: small enough to force a compressed representation, comfortably wider
//--- than the decision itself. The floor below covers the regression head, where 4x1 would be absurd.
refactor(ai): derive the dense taper's shape, not just its first layer Deriving the first layer's width left NeuronsReduction and MinNeuronsCount behind as inputs calibrated for something that no longer exists. Against a hand-picked 500-wide first layer "keep 30%, floor at 20" produced a genuine funnel - 500 -> 150 -> 45. Against the derived 64 it degenerates to 64 -> 20 -> 20: the reduction factor stops mattering after one step, and "minimum neurons per layer" silently becomes the width of every layer but the first. Two knobs whose labels no longer describe what they do. The taper now runs geometrically from the derived first-layer width down to a final hidden layer sized off the output count, spread evenly over however many layers the chosen AIType implies: MLP_3L 64 -> 28 -> 12 -> 3 29,151 dense weights MLP_4L 64 -> 37 -> 21 -> 12 -> 3 30,450 CONV/LSTM/HYBRID_2L 64 -> 12 -> 3 27,763 and it stays a funnel at the floor, where the old rule could not: D1 (first layer floored to 16) 16 -> 14 -> 12 -> 3 Both inputs are removed. With the width derived there is no freedom left in the taper, so keeping either would only let the user contradict the derivation. The layer COUNT stays selectable, because it is bundled into AIType alongside the conv/LSTM front-end - depth is an architecture choice, not a data-derived quantity, and pairing them means the two cannot contradict each other. m_minNeuronsCount / m_neuronsReduction survive as frozen members: nothing reads them to build a topology any more, but they hold positional slots in the .cfg sidecar and the weights fingerprint, and changing either value would re-key every model on disk for no behavioural reason. The DB config fingerprint drops both terms. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 14:03:42 -04:00
#define HIDDEN_TAPER_OUTPUT_MULTIPLE 4
//--- 8, NOT the 20 from the MQL5 "4 hidden layers" article - that floor is load-bearing on ITS 1000-wide
//--- first layer, and this codebase MEASURES the first layer instead (16 units on live SP500 H4). At 16
//--- a floor of 20 makes lastHidden >= m_initialNeuronsCount, so the WIDTH TAPER NEVER RUNS.
feat: derived taper restored; DB ranking reads a reserved slice, shrunk TOPOLOGY - reverts the two constants and drops CausalHiddenLayerFloor. The MQL5 article's 30%-per-layer cut and floor of 20 are load-bearing on ITS first-layer width of 1000 (1000->300->90->27 needs a floor to stop). This codebase MEASURES that width, and on the live SP500 H4 config it is 16 units - already floored, with the budget printing "11360 estimated in-sample bars cannot support a 800-wide input ... roughly 1.1 weights per training bar - expect overfitting". At 16 units a floor of 20 makes lastHidden >= m_initialNeuronsCount, so ComputeHiddenLayerCount returns on its first branch and the width taper - the only part derived from this symbol's data - became dead code on all four ensemble members, with depth (2 -> 4) set entirely by counting feature domains. ComputeLayerWidths had already rejected this exact pair of constants in its own comment. The causal floor's premise does not hold either: layers are not inference steps. The "1 layer linear / 2 nonlinear / 3 multi-connected" result is Lippmann 1987 and is about hard-threshold units; with sigmoid/ReLU, Cybenko 1989 and Hornik 1991 give universal approximation from a single hidden layer. Depth buys parameter efficiency for compositional functions, not reasoning hops. ForceHiddenLayers remains for measuring depth directly. RANKING SLICE - the backfill no longer reads the window it is judged on. The deployed checkpoint is CHOSEN as the best-scoring era on the OOS window, so win rates measured back over it are selection-inflated, and the backfill was writing exactly those into the table filter weights rank on: the selection set consumed twice, beside a deploy gate that applies a Sidak correction for that effect. The newest RANK_SLICE_PCT_OF_OOS (20%) of the OOS window, plus a label-horizon purge, is now reserved and graded by nothing - not pass 3, not checkpoint selection, not the gate. The backfill reads only that. The gate keeps ~80% of its measurement (power goes as the square root, so ~10% of a sigma), and the slice is the newest data, which is the regime about to be traded. RankSliceBars returns 0 when no honest slice fits and the backfill then REFUSES and says so, rather than falling back to the scoring window and looking like a success. SHRINKAGE - per-tier win rates are shrunk toward the filter's own pooled rate by MIN_TRADES_FOR_WIN_RATE pseudo-trades before becoming weights. The raw ratio at the minimum sample count carries a ~15pp standard error, so a tier that went 8-2 was handed weight 80 and outranked a tier measured over hundreds of calls at 55 - the ranking was being driven by which small tier got lucky. Opt-in per call site (priorWeight 0 keeps the raw behaviour). Compile-verified: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 21:49:52 -04:00
#define HIDDEN_TAPER_MIN_WIDTH 8
//--- ComputeHiddenLayerCount() bounds. 2.0 rather than the article's 10/3: halving still produces a
//--- taper at the widths derived here.
feat: derived taper restored; DB ranking reads a reserved slice, shrunk TOPOLOGY - reverts the two constants and drops CausalHiddenLayerFloor. The MQL5 article's 30%-per-layer cut and floor of 20 are load-bearing on ITS first-layer width of 1000 (1000->300->90->27 needs a floor to stop). This codebase MEASURES that width, and on the live SP500 H4 config it is 16 units - already floored, with the budget printing "11360 estimated in-sample bars cannot support a 800-wide input ... roughly 1.1 weights per training bar - expect overfitting". At 16 units a floor of 20 makes lastHidden >= m_initialNeuronsCount, so ComputeHiddenLayerCount returns on its first branch and the width taper - the only part derived from this symbol's data - became dead code on all four ensemble members, with depth (2 -> 4) set entirely by counting feature domains. ComputeLayerWidths had already rejected this exact pair of constants in its own comment. The causal floor's premise does not hold either: layers are not inference steps. The "1 layer linear / 2 nonlinear / 3 multi-connected" result is Lippmann 1987 and is about hard-threshold units; with sigmoid/ReLU, Cybenko 1989 and Hornik 1991 give universal approximation from a single hidden layer. Depth buys parameter efficiency for compositional functions, not reasoning hops. ForceHiddenLayers remains for measuring depth directly. RANKING SLICE - the backfill no longer reads the window it is judged on. The deployed checkpoint is CHOSEN as the best-scoring era on the OOS window, so win rates measured back over it are selection-inflated, and the backfill was writing exactly those into the table filter weights rank on: the selection set consumed twice, beside a deploy gate that applies a Sidak correction for that effect. The newest RANK_SLICE_PCT_OF_OOS (20%) of the OOS window, plus a label-horizon purge, is now reserved and graded by nothing - not pass 3, not checkpoint selection, not the gate. The backfill reads only that. The gate keeps ~80% of its measurement (power goes as the square root, so ~10% of a sigma), and the slice is the newest data, which is the regime about to be traded. RankSliceBars returns 0 when no honest slice fits and the backfill then REFUSES and says so, rather than falling back to the scoring window and looking like a success. SHRINKAGE - per-tier win rates are shrunk toward the filter's own pooled rate by MIN_TRADES_FOR_WIN_RATE pseudo-trades before becoming weights. The raw ratio at the minimum sample count carries a ~15pp standard error, so a tier that went 8-2 was handed weight 80 and outranked a tier measured over hundreds of calls at 55 - the ranking was being driven by which small tier got lucky. Opt-in per call site (priorWeight 0 keeps the raw behaviour). Compile-verified: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 21:49:52 -04:00
#define HIDDEN_TAPER_TARGET_RATIO 2.0
feat(nn): derive dense depth, train on all history, pin the shape in .cfg Completes the derived-topology work. Three inputs removed. AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five entries instead of eight. Depth is now derived from the two endpoints the taper already has to connect (derived first-layer width, output-tied final width) at a 2x per-layer compression target, clamped [2..5]. Asking a user to pick a layer count while the code derives the widths those layers taper between was asking for half a decision: at 64 units tapering to 12, four layers compress by 1.4x per step and five by 1.3x, so the extra depth bought no abstraction. On the shipping H1/10y default the derivation lands on 3 layers - the depth that actually won Run 2. StudyPeriods removed. There is no case for training on less data than the broker provides at a ~6% directional base rate; the honest generalization read comes from the OOS holdout, not from withholding history. Training now starts at the earliest available bar, floored by MinTrainYear, which answers a different question (excluding dubious pre-history) and stays. That required closing the hazard the old code documented: the capacity budget now MEASURES the symbol's real bar count, and a topology derived from a measurement would widen as history downloads. Both ends are now pinned. Every derived value left the weights-filename fingerprint - keying a filename on a measured quantity means the EA looks for a file that does not exist, starts from era 0 and orphans a trained model, silently, because a missing cache is the normal first-run state. The shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the four derived fields rather than diffing them; a mismatch there would discard a fully-trained model over nothing the user did. Two fields appended to the .cfg for the conv/LSTM stages, length-guarded on read because FileReadInteger past EOF returns 0 with no error. ForceHiddenLayers, a compile-time constant like DebuggingMode, pins depth for diagnostic comparisons. It joins the fingerprint only when non-zero, so forced depths get their own files - sequential comparisons only, not simultaneous from one .ex5. Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64, 3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from ~58k to ~28k weights. Both builds compile 0 errors, 0 warnings. Re-keys existing models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
#define MIN_HIDDEN_LAYERS 2
#define MAX_HIDDEN_LAYERS 5
//--- EstimatedInSampleBars() fallback while history is still downloading - below the trusted-bar floor
//--- the measurement says more about the sync state than about the symbol.
feat(nn): derive dense depth, train on all history, pin the shape in .cfg Completes the derived-topology work. Three inputs removed. AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five entries instead of eight. Depth is now derived from the two endpoints the taper already has to connect (derived first-layer width, output-tied final width) at a 2x per-layer compression target, clamped [2..5]. Asking a user to pick a layer count while the code derives the widths those layers taper between was asking for half a decision: at 64 units tapering to 12, four layers compress by 1.4x per step and five by 1.3x, so the extra depth bought no abstraction. On the shipping H1/10y default the derivation lands on 3 layers - the depth that actually won Run 2. StudyPeriods removed. There is no case for training on less data than the broker provides at a ~6% directional base rate; the honest generalization read comes from the OOS holdout, not from withholding history. Training now starts at the earliest available bar, floored by MinTrainYear, which answers a different question (excluding dubious pre-history) and stays. That required closing the hazard the old code documented: the capacity budget now MEASURES the symbol's real bar count, and a topology derived from a measurement would widen as history downloads. Both ends are now pinned. Every derived value left the weights-filename fingerprint - keying a filename on a measured quantity means the EA looks for a file that does not exist, starts from era 0 and orphans a trained model, silently, because a missing cache is the normal first-run state. The shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the four derived fields rather than diffing them; a mismatch there would discard a fully-trained model over nothing the user did. Two fields appended to the .cfg for the conv/LSTM stages, length-guarded on read because FileReadInteger past EOF returns 0 with no error. ForceHiddenLayers, a compile-time constant like DebuggingMode, pins depth for diagnostic comparisons. It joins the fingerprint only when non-zero, so forced depths get their own files - sequential comparisons only, not simultaneous from one .ex5. Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64, 3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from ~58k to ~28k weights. Both builds compile 0 errors, 0 warnings. Re-keys existing models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
#define TOPOLOGY_BUDGET_MIN_TRUSTED_BARS 500
#define TOPOLOGY_BUDGET_FALLBACK_YEARS 10
//--- How much the conv stage compresses one bar's feature vector. The layer is a per-bar projection, so
//--- filters > features EXPANDS a correlated input at the very bottom of the stack.
feat(nn): derive conv filter count and LSTM hidden size from the data Same defect the first-layer width had before 2026-07-29: both were inputs whose defaults were fixed constants picked with no reference to the input they sit on, which is the only thing that decides whether either number is sane. The conv layer is a per-bar projection - AddConvStage sets window = step = one bar's features - so its filter count should be read against the per-bar feature count. Sixteen filters COMPRESSED a 50-feature configuration 3x but EXPANDED a minimal 4-feature one 4x, and the expanding case adds parameters below every learnable layer without adding information. Now derived as half the per-bar feature count, snapped down a power-of-two ladder. The LSTM stage was the bigger miss. Its weight count is exactly 4*H*(H+inputs+1) (CNeuronLSTMOCL::SetInputs) and AddLstmStage feeds it the whole flattened vector, so the shipped 32 units against a 540-wide input is ~73k weights - more than DOUBLE the entire derived dense taper it feeds. It was the one stage the capacity budget never covered, which is why deriving the dense stack alone did not stop LSTM and HYBRID from being over-parameterized. Now solved from the same one-weight-per-in-sample-bar budget the first layer spends. Factored EstimatedInSampleBars() out of ComputeFirstLayerWidth so all three decisions spend one budget rather than each guessing at the training-set size separately. Both new values are assigned alongside the first-layer width, before the fingerprint that hashes them, and are functions of inputs already in that hash - so they need no entry of their own, and the same reasoning removes them from the DB config key. Both builds compile 0 errors, 0 warnings. Re-keys existing models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 09:22:11 -04:00
#define CONV_COMPRESSION_DIVISOR 2
#define CONV_FILTERS_MIN 4
#define CONV_FILTERS_MAX 32
#define LSTM_HIDDEN_MIN 8
#define LSTM_HIDDEN_MAX 128
//--- Balanced accuracy of a model that puts every bar in ONE class - the FLOOR of the metric, not a
//--- midpoint. Any genuinely multi-class model scores above it.
fix(training): escape the recall-gate catch-22 that let runs decay unchecked Evidence (MQL5\Logs, SP500 H1, 2026-07-29): Perceptron era 61 Buy 32% Sell 27% Neut 94% bal 51% LSTM era 160 Buy 16% Sell 11% Neut 98% bal 42% (peaked 49% @ era 44) Hybrid era 179 Buy 5% Sell 2% Neut 99% bal 35% (peaked 41%) CONV era 228 Buy 2% Sell 4% Neut 99% bal 35% (peaked 40% @ era 122) Every model peaks early then decays monotonically toward Neutral, and nothing stops it: the restore-best-weights + decay-eta handler is gated on m_bestPassedRecall, which stays false forever when no checkpoint ever clears the per-class floor. CONV ran 228 eras with eta pinned at its 0.000300 start. The plateau ladder cannot end such a run either (stage 3 refuses to deploy without a recall pass, so it resets ~27 times), making it a 1000-era one-way trip. The gate's own justification had expired. It was written when the pre-pass tiebreak was blended-accuracy-only, where "best" really did mean "called Neutral most confidently". The balanced-selection change replaced that with `balancedOosEra > m_bestBalancedOos` plus an isFullyCollapsedEra exclusion, so a Neutral-only era now scores ~33% - the FLOOR of the balanced metric - and cannot anchor the checkpoint at all. Pre-pass "best" now means "most class-balanced so far", which is worth defending; and isWorseEra is itself a balanced-accuracy regression, so it cannot fire merely for trading Neutral calls for Buy/Sell. The original concern still holds while the best-so-far IS near-collapse, so the escape is margin-guarded: defend the checkpoint only once balanced accuracy sits more than BALANCED_WORTH_DEFENDING_MARGIN_PCT (5pp) above the one-class floor of 100/3. Against the run above that engages for all three stuck topologies (42.3/41.3/50.0 vs a 38.3 threshold) while a genuinely collapsed run still explores freely. Two inputs restored to the regime that actually produced a deploy: - MinRecall 60 -> 40. The one successful auto-deploy in the logs (Hybrid, 28th 00:50, best balanced 66.0%) ran against a 40% floor. 60 has never been shown reachable here - a floor above what the config can reach is the same "target set too high" failure the surrounding comment already warns about. - OversampleParity 60 -> 90. 60 overcorrected. Runs now START Neutral-dominant (Buy 0-11% recall at era 1) and call Buy/Sell on 0-4% of bars against a ~6% true base rate - under-calling, with no headroom to converge down from. The deploying run began at Buy 90% / Sell 36%, 24% of bars called, and settled into the floor from above. Raw over-calling is the intended starting condition; live calls are base-rate-calibrated by AILogitPriorStrength, which is why the input's own note says to judge over-calling by live-fired precision, not raw counts. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 07:51:08 -04:00
#define BALANCED_COLLAPSE_PCT (100.0 / 3.0)
//--- How far above that floor a best-so-far checkpoint must sit before the regression handler defends
//--- it: "still basically a collapse, keep exploring" versus "a real multi-class state we are sliding
//--- off", the case that ran unchecked for 228 eras.
fix(training): escape the recall-gate catch-22 that let runs decay unchecked Evidence (MQL5\Logs, SP500 H1, 2026-07-29): Perceptron era 61 Buy 32% Sell 27% Neut 94% bal 51% LSTM era 160 Buy 16% Sell 11% Neut 98% bal 42% (peaked 49% @ era 44) Hybrid era 179 Buy 5% Sell 2% Neut 99% bal 35% (peaked 41%) CONV era 228 Buy 2% Sell 4% Neut 99% bal 35% (peaked 40% @ era 122) Every model peaks early then decays monotonically toward Neutral, and nothing stops it: the restore-best-weights + decay-eta handler is gated on m_bestPassedRecall, which stays false forever when no checkpoint ever clears the per-class floor. CONV ran 228 eras with eta pinned at its 0.000300 start. The plateau ladder cannot end such a run either (stage 3 refuses to deploy without a recall pass, so it resets ~27 times), making it a 1000-era one-way trip. The gate's own justification had expired. It was written when the pre-pass tiebreak was blended-accuracy-only, where "best" really did mean "called Neutral most confidently". The balanced-selection change replaced that with `balancedOosEra > m_bestBalancedOos` plus an isFullyCollapsedEra exclusion, so a Neutral-only era now scores ~33% - the FLOOR of the balanced metric - and cannot anchor the checkpoint at all. Pre-pass "best" now means "most class-balanced so far", which is worth defending; and isWorseEra is itself a balanced-accuracy regression, so it cannot fire merely for trading Neutral calls for Buy/Sell. The original concern still holds while the best-so-far IS near-collapse, so the escape is margin-guarded: defend the checkpoint only once balanced accuracy sits more than BALANCED_WORTH_DEFENDING_MARGIN_PCT (5pp) above the one-class floor of 100/3. Against the run above that engages for all three stuck topologies (42.3/41.3/50.0 vs a 38.3 threshold) while a genuinely collapsed run still explores freely. Two inputs restored to the regime that actually produced a deploy: - MinRecall 60 -> 40. The one successful auto-deploy in the logs (Hybrid, 28th 00:50, best balanced 66.0%) ran against a 40% floor. 60 has never been shown reachable here - a floor above what the config can reach is the same "target set too high" failure the surrounding comment already warns about. - OversampleParity 60 -> 90. 60 overcorrected. Runs now START Neutral-dominant (Buy 0-11% recall at era 1) and call Buy/Sell on 0-4% of bars against a ~6% true base rate - under-calling, with no headroom to converge down from. The deploying run began at Buy 90% / Sell 36%, 24% of bars called, and settled into the floor from above. Raw over-calling is the intended starting condition; live calls are base-rate-calibrated by AILogitPriorStrength, which is why the input's own note says to judge over-calling by live-fired precision, not raw counts. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 07:51:08 -04:00
#define BALANCED_WORTH_DEFENDING_MARGIN_PCT 5.0
//--- PLATEAU LADDER. So: count eras since the last new best and escalate. ANY new best resets
//--- counter and stage.
#define PLATEAU_PATIENCE_ERAS 8 // eras with no new best balanced accuracy before escalating a stage
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
#define PLATEAU_STAGE_RESTART 1 // first boosted warm restart (see PLATEAU_RESTART_BOOST)
#define PLATEAU_STAGE_ANNEAL 2 // second boosted warm restart (the gamma anneal it named is gone)
#define PLATEAU_STAGE_DEPLOY 3 // exhausted: deploy the best checkpoint and finish the run
//--- Restart amplitude. Escaping a basin needs a rate LARGER than the one that settled into it;
//--- SGDR restarts span 10-100x, this is tamer because MAX_WEIGHT_DELTA and the checkpoint restore
//--- already bound the blast radius.
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
#define PLATEAU_RESTART_BOOST 5.0
//--- IN-SAMPLE EARLY STOP. Training error is noisy per era - mini-batch order alone moves it - and
//--- ending a run that is still learning costs far more than a few wasted eras.
feat(search): stop on the IN-SAMPLE plateau, and shrink every best-of-K effect before quoting it Points 3 and 4 of the four-point plan. 1. IN-SAMPLE EARLY STOP - and the reason it is worth having is not compute. The plateau ladder stops on the OOS SELECTION score. That is a peek: by the time it fires, every one of those eras has been evaluated out of sample, so all of them sit in the family the deploy gate corrects over (g_ensCandidateEras, Sidak). Training longer therefore does not merely cost time - it RAISES the bar the eventual winner has to clear. The new stop reads the TRAINING error, which the gate never looks at. When the optimiser has stopped improving on data it can see, more eras will not find a better model; they will only enlarge the OOS family. Ending there shrinks the correction, and the shrinkage is legitimate precisely BECAUSE the stopping rule never consulted an out-of-sample number. That distinction is the whole point and it is the one this project has got wrong four times: stop on IS and the family really is smaller; stop on OOS and those eras were searched and still count. Both stops now exist; only this one buys a lower bar. Deliberately more patient than the OOS ladder (IS_ERROR_PATIENCE_MULT = 3x): training error is noisy per era - mini-batch order alone moves it - and ending a run that is still learning costs far more than a few wasted eras. Improvement is RELATIVE (IS_ERROR_IMPROVE_FRAC = 1%), so it does not depend on the loss's absolute scale, and it only acts when a checkpoint exists, since otherwise it would end a run with nothing to deploy. Reset per RUN alongside the ladder, so a resumed run cannot early-stop on its first era against a previous run's best. 2. WINNER'S-CURSE SHRINKAGE ON THE BARRIER-GEOMETRY WINNER. The family-wise permutation gate already establishes that the RANKING is not noise. It says nothing about the SIZE of the winner's effect - and a best-of-K maximum is biased upward by construction, being the largest of K noisy draws. The adoption message quotes that raw maximum and compares it against the incumbent, so the number a reader plans on is the inflated one. The penalty is now measured, not assumed: the same permutation draws that produce the p-value also produce, per draw, the MAXIMUM excess across all candidates under pure noise. The mean of those maxima is exactly what a best-of-K selection is expected to report when there is nothing there. This is the empirical form of the sqrt(2 ln K) x SE penalty the SQX EdgeFinder plugin applies to every maximum it reports (Stats.java:79-88), and it needs no normality assumption because the draws ARE the null distribution. Applied in James-Stein form - effect x max(0, 1 - penalty^2/effect^2) - so a large effect is nearly untouched and a marginal one collapses toward zero. Reported, not gated. The adoption decision still turns on the permutation p-value, which is the right test for "is the ranking real"; the shrunk number is there so the magnitude quoted beside it is one worth planning on. Closes the first of the two EdgeFinder ports identified on 2026-08-12. NOTE on the second EdgeFinder port, deliberately not done here: "let the measurement steer the target" is already true where it matters most - ReportGeometryExpectancyScan ADOPTS the winning barrier geometry under the family-wise gate rather than advising it, and the MI excursion suite publishes a verdict per instrument per config. What is still missing is steering the TRAINING TARGET itself (direction vs excursion) off those verdicts, and that is a design change rather than a surgical one - direction is a closed verdict while excursion SIZE keeps clearing, so the honest version of that change is a target-selection policy, not a flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:29:00 -04:00
#define IS_ERROR_IMPROVE_FRAC 0.01
#define IS_ERROR_PATIENCE_MULT 3
//--- FILE-COMPATIBILITY SHIMS for three removed inputs. So the old default is still written and
//--- hashed, and the .cfg field is no longer COMPARED on load - a model saved under any previous
//--- value still loads.
#define LEGACY_CONVERGE_WR_SLOT 80
feat(nn): derive dense depth, train on all history, pin the shape in .cfg Completes the derived-topology work. Three inputs removed. AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five entries instead of eight. Depth is now derived from the two endpoints the taper already has to connect (derived first-layer width, output-tied final width) at a 2x per-layer compression target, clamped [2..5]. Asking a user to pick a layer count while the code derives the widths those layers taper between was asking for half a decision: at 64 units tapering to 12, four layers compress by 1.4x per step and five by 1.3x, so the extra depth bought no abstraction. On the shipping H1/10y default the derivation lands on 3 layers - the depth that actually won Run 2. StudyPeriods removed. There is no case for training on less data than the broker provides at a ~6% directional base rate; the honest generalization read comes from the OOS holdout, not from withholding history. Training now starts at the earliest available bar, floored by MinTrainYear, which answers a different question (excluding dubious pre-history) and stays. That required closing the hazard the old code documented: the capacity budget now MEASURES the symbol's real bar count, and a topology derived from a measurement would widen as history downloads. Both ends are now pinned. Every derived value left the weights-filename fingerprint - keying a filename on a measured quantity means the EA looks for a file that does not exist, starts from era 0 and orphans a trained model, silently, because a missing cache is the normal first-run state. The shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the four derived fields rather than diffing them; a mismatch there would discard a fully-trained model over nothing the user did. Two fields appended to the .cfg for the conv/LSTM stages, length-guarded on read because FileReadInteger past EOF returns 0 with no error. ForceHiddenLayers, a compile-time constant like DebuggingMode, pins depth for diagnostic comparisons. It joins the fingerprint only when non-zero, so forced depths get their own files - sequential comparisons only, not simultaneous from one .ex5. Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64, 3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from ~58k to ~28k weights. Both builds compile 0 errors, 0 warnings. Re-keys existing models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
#define LEGACY_STUDY_PERIOD_SLOT 0
#define LEGACY_HISTORY_BARS_SLOT 20
//--- Derived-window rule: median confirmed swing leg, snapped DOWN to the ladder, capped.
#define HISTORY_BARS_FALLBACK 20
#define HISTORY_BARS_FLOOR 12
#define WINDOW_DERIVE_SPAN_BARS 20000
#define WINDOW_DERIVE_MIN_LEGS 30
#define WINDOW_SWING_WING 12
//--- True OOS samples a class needs before its recall is trusted as a real pass - enough to rule out the
//--- zero-sample degenerate case that produced a false convergence at eras 44-46.
#define MIN_OOS_CLASS_SAMPLES_FOR_GATE 10
//--- A class this rare cannot carry the per-class recall floor. NEUTRAL ONLY - the directional
//--- classes are deliberately not exempted by prevalence.
fix: the recall gate was unsatisfiable and the LR decay was a spiral Both made the run structurally unable to succeed, independently of any signal in the data. Found by reading the 13:01 log. RECALL GATE. m_objectiveMet required Buy, Sell AND Neutral OOS recall each >= 40%. First-touch resolution (ce52654) collapsed Neutral from the ~94% majority it was under exact-pivot labels to a same-bar-tie residue - 250 of 38,261 bars, 0.65% - so the floor was asking the model to identify 40% of coin-flip ties before it could converge. Measured: CONV, LSTM and HYBRID all logged "Neutral:0% (need >=40% each)" on every era. No model could ever satisfy it; every run was destined for the plateau ladder or the era cap. Only the DIRECTIONAL floors are load-bearing for the anti-collapse job the gate exists to do: an all-Neutral model shows Buy and Sell recall at 0% and is blocked by them. Neutral's own floor guarded the mirror bias (over-calling Buy/Sell at Neutral's expense), which was real at 94% prevalence and is not at 0.65% - there, almost never calling Neutral is correct rather than biased. Prevalence-guarded rather than hardcoded off, so it returns by itself if a future label rule makes Neutral substantial again. Deliberately NOT extended to Buy/Sell: exempting a thin directional class reopens the era-44-46 hole, which directionalRecallMeasured only half-covers - it checks those classes were MEASURED, not that they passed. ETA DECAY. A regressing era restored the checkpoint, reset the optimizer and cut eta - all on the FIRST regression. The next era then started from an identical state with a smaller step, regressed again, and got the same treatment. The loop is self-sustaining and cannot discover anything, because rolling the weights back is exactly what removes the exploration that would end it. Measured on PAI: eras 2-11 every one a regression against era 1, eta 0.000594 -> 0.000024, dW/W 0.000%/0.000% from era 2 onward. Ten eras, ~45s each, reproducing era 1 exactly and unable to do anything else. Now requires ETA_DECAY_PATIENCE_ERAS consecutive regressions - the standard ReduceLROnPlateau formulation. A single bad era is noise, and an improving era clears the counter so alternating runs never accumulate into a decay. Build tag -> gate-patience-v3. It had not moved in six commits, which is why the running binary could not be identified from its own log. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:28:58 -04:00
#define MIN_GATE_CLASS_SHARE_PCT 5.0
//--- Consecutive regressing eras before the checkpoint is restored and g_eta decayed. Patience is
//--- the standard ReduceLROnPlateau formulation and restores that exploration.
fix: the recall gate was unsatisfiable and the LR decay was a spiral Both made the run structurally unable to succeed, independently of any signal in the data. Found by reading the 13:01 log. RECALL GATE. m_objectiveMet required Buy, Sell AND Neutral OOS recall each >= 40%. First-touch resolution (ce52654) collapsed Neutral from the ~94% majority it was under exact-pivot labels to a same-bar-tie residue - 250 of 38,261 bars, 0.65% - so the floor was asking the model to identify 40% of coin-flip ties before it could converge. Measured: CONV, LSTM and HYBRID all logged "Neutral:0% (need >=40% each)" on every era. No model could ever satisfy it; every run was destined for the plateau ladder or the era cap. Only the DIRECTIONAL floors are load-bearing for the anti-collapse job the gate exists to do: an all-Neutral model shows Buy and Sell recall at 0% and is blocked by them. Neutral's own floor guarded the mirror bias (over-calling Buy/Sell at Neutral's expense), which was real at 94% prevalence and is not at 0.65% - there, almost never calling Neutral is correct rather than biased. Prevalence-guarded rather than hardcoded off, so it returns by itself if a future label rule makes Neutral substantial again. Deliberately NOT extended to Buy/Sell: exempting a thin directional class reopens the era-44-46 hole, which directionalRecallMeasured only half-covers - it checks those classes were MEASURED, not that they passed. ETA DECAY. A regressing era restored the checkpoint, reset the optimizer and cut eta - all on the FIRST regression. The next era then started from an identical state with a smaller step, regressed again, and got the same treatment. The loop is self-sustaining and cannot discover anything, because rolling the weights back is exactly what removes the exploration that would end it. Measured on PAI: eras 2-11 every one a regression against era 1, eta 0.000594 -> 0.000024, dW/W 0.000%/0.000% from era 2 onward. Ten eras, ~45s each, reproducing era 1 exactly and unable to do anything else. Now requires ETA_DECAY_PATIENCE_ERAS consecutive regressions - the standard ReduceLROnPlateau formulation. A single bad era is noise, and an improving era clears the counter so alternating runs never accumulate into a decay. Build tag -> gate-patience-v3. It had not moved in six commits, which is why the running binary could not be identified from its own log. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:28:58 -04:00
#define ETA_DECAY_PATIENCE_ERAS 3
//--- Bound on FindConfirmedZigZagPivot()'s backward scan. Generous rather than tight: the scan is plain
//--- array reads and its result is cached per bar, so a long scan is paid at most once per unique bar.
#define SWING_SCAN_CAP_BARS 750
//--- EMA shadow-weight deployment blend rate - see m_shadowNet. 0.01 matches the Tau range used for
//--- target-network soft updates in the Gizlyk reference RL algorithms: small enough that no single
//--- era's raw weights move the deployed model far, large enough to track sustained learning.
#define SHADOW_WEIGHT_TAU 0.01
//--- MINI-BATCH SIZE. 1 restores the exact per-sample SGD this engine had until 2026-08-09, and
//--- every helper below is an identity there. Fewer steps need a larger step - sqrt(B) for adaptive
//--- methods (Krizhevsky 2014; Granziol et al.
fix: the Adam second moment was never Adam - all four tiers Root cause of the B=32 regression, and it predates F4 entirely. Every Adam kernel stored v already square-rooted and then fed that stored value back in as if it were the variance: v_new = sqrt(b2 * v_old + (1 - b2) * g^2) That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below unit scale, so the denominator stops tracking the gradient and Adam degrades into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll (batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where a scale-invariant optimizer gives the same distance for both. After the fix all six magnitudes read 1.199 and v tracks |g| exactly. It hit conv/LSTM specifically because they sit behind a batch-norm with running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep in the degraded regime - while the dense stack near the loss stayed in the working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/ 0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already squared v back for gamma/beta and its comment named the kernels as wrong, which is exactly why gamma/beta kept training while the stages behind froze. Persisted .nnw needs no migration - v keeps its std-dev meaning. Also, the two ways F4 exposed it, both mine: - No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods (Krizhevsky 2014; Granziol et al. 2022), applied once in InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD. - Plateau patience denominated in eras, so raising B made the ladder 32x more impatient in its only unit. PAI converged at era 41 on ~49k updates where the same config had been finding new bests at era 1028. TrainPlateauPatienceEras() stretches it by the same sqrt(B). TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23 eras per stage, not 8 -> 45). Both helpers are identities at B=1. Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into the ranking key rather than checked at deploy time so a one-sided era cannot become best-so-far in the first place. Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded CAppDialog teardown that sat ahead of it - the same ordering inversion the rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit (vs ~1.1 s for the three that finished) having reached none of its cleanup, so its arrows stayed on the chart. Steps are now timed in the log. PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now walks every object type and reports the object counts when both are zero. Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt. FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
#define TRAIN_BATCH_SIZE 8
//--- Floor for Train()'s indicator-depth clamp. Above it a short-but-real history beats livelocking on a
//--- depth the terminal will never serve; below it a tiny BarsCalculated() is more likely an indicator
//--- mid-calculation than a hard cap, so the clamp stands down. Sized so a clamped era still holds an
//--- OOS window worth measuring.
fix(train): clamp the sweep to indicator-servable depth - the scan wall was CopyBuffer, not a cold indicator Symptom: on a 3-chart run with contention ruled out (SP500 sitting at era 2552), USDJPY and XAUUSD produced 0 usable windows out of 50,163 and 33,966 - forever, re-sweeping on every discard, which is the panel oscillating 0->100%. Bars() is the PRICE series depth. A CUSTOM indicator's is not: MT5 calculates it in its own context bounded by "Max bars in chart" (TERMINAL_MAXBARS), and CopyBuffer past that limit does not short-read, it FAILS - so CDoubleBuffer keeps nothing and EVERY index answers EMPTY_VALUE. ADMovingAverage is the only custom indicator whose feature block REJECTS on EMPTY_VALUE (ADZigZag, also CiCustom, neutral-fills; RSI/MACD/Ichimoku/ATR are built-ins served at any depth), so the sweep died on feature 25 of every bar while the 24 price features under it were fine. That is exactly the "window had 24 of 832 values" the stall report named. Perfectly depth-correlated, measured 2026-08-17: SP500 16,234 bars -> era 2552 XAUUSD 33,982 -> 0 windows XTIUSD 16,611 bars -> era 71 USDJPY 50,179 -> 0 windows This RETIRES the 2026-08-17 cold-indicator reading of the same stall. f0cf659 was right that the rejection must be transient and that the dead backoff had to arm - the branch did change to 'cold-indicator backoff' - but waiting cannot fix a depth the terminal will never grant. So Train() now clamps to TunableBarsCalculated() (which existed and was only ever used for a tuner printout) and trains on the history that IS available, naming TERMINAL_MAXBARS in the log so the cause is readable next time. m_coldSweepTick still owns the genuinely transient case: that reads back as -1, not a positive short count. Recomputed per era, so the clamp lifts by itself if the setting is raised. Also fixes a real off-by-one it was hiding: the MA block reads GetData(idx) AND GetData(idx + 1) for its bar-over-bar change, but ResizeBuffers sized m_MA to barIndex exactly - so the deepest bar of every sweep read one past the end and was rejected as cold. Same shape as the +ichiKijun the Ichimoku/close pair already has. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:27:14 -04:00
#define TRAIN_MIN_CLAMPED_BARS 2000
//--- SettledBars()'s wait. Long enough that a busy terminal makes visible progress between probes, short
//--- enough that a chart with nothing to wait for loses only seconds.
feat(depth): prime -> settle -> sweep, and name which handle is short "Max bars in chart" is set to Unlimited, so the static-terminal-limit reading in 1dda479 was wrong. Two other candidate causes are falsified too: the price series is fully downloaded and flat (USDJPY 50,162 -> 50,163 over 80 minutes, i.e. one new H4 bar), and the handles have been stable since 13:07 with zero windows for the 17 minutes after, so it is not download-in-progress and not handle churn. The MA period tops out at 200 (ADIndicatorTuner MA_PERIOD_PRESETS) against ~50k bars, so it is not indicator cost either. What IS verified stays verified: CopyBuffer past the calculated depth fails outright rather than short-reading, so the buffer holds nothing and every index reads EMPTY_VALUE; m_MA is the only CiCustom whose block REJECTS on that (m_ADZigZag neutral-fills, RSI/MACD/Ichimoku/ATR are built-ins); the wall is therefore feature 25 of every bar, exactly as the "24 of 832" stall lines said. And it is depth-correlated: 16k-bar charts train, 34k/50k get zero windows forever. So the WHY is still open, and this fix does not depend on it. Per the user's protocol: the request itself is the primer, so prime at full depth, then poll TunableBarsCalculated() every 3s and hold the sweep until it stops changing (3 steady probes), then use whatever it settled at. Bounded at 10 min, and a give-up is logged as a give-up so an abandoned depth is never mistaken for a settled one. This supersedes 1dda479's clamp on the two training paths, which snapshotted a value that may still have been climbing; the clamp remains for the paths that cannot wait (inference/online/rescan/export, see 7e63a8b). The load-bearing part is what does NOT happen while waiting: no sweep. A 50k-bar feature scan starves the indicator threads the request just woke, which is how the failure sustained itself for 40 minutes at a time - discard era, re-sweep, discard, which is the 0->100% oscillation on the panel. Also adds IndicatorDepthReport(): per-handle BarsCalculated() on the priming, cap and stall lines. The logs proved WHICH FEATURE died but never WHICH HANDLE was short, so the cause had to be inferred - and was guessed wrong twice. The next occurrence reads it off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:44:27 -04:00
#define DEPTH_SETTLE_PROBE_MS 3000
#define DEPTH_SETTLE_STABLE_PROBES 3
//--- Hard stop. A depth that has not settled in 10 minutes is not going to, and training on the history
//--- that IS there beats waiting forever - the give-up is logged, so it is never confused with a settle.
feat(depth): prime -> settle -> sweep, and name which handle is short "Max bars in chart" is set to Unlimited, so the static-terminal-limit reading in 1dda479 was wrong. Two other candidate causes are falsified too: the price series is fully downloaded and flat (USDJPY 50,162 -> 50,163 over 80 minutes, i.e. one new H4 bar), and the handles have been stable since 13:07 with zero windows for the 17 minutes after, so it is not download-in-progress and not handle churn. The MA period tops out at 200 (ADIndicatorTuner MA_PERIOD_PRESETS) against ~50k bars, so it is not indicator cost either. What IS verified stays verified: CopyBuffer past the calculated depth fails outright rather than short-reading, so the buffer holds nothing and every index reads EMPTY_VALUE; m_MA is the only CiCustom whose block REJECTS on that (m_ADZigZag neutral-fills, RSI/MACD/Ichimoku/ATR are built-ins); the wall is therefore feature 25 of every bar, exactly as the "24 of 832" stall lines said. And it is depth-correlated: 16k-bar charts train, 34k/50k get zero windows forever. So the WHY is still open, and this fix does not depend on it. Per the user's protocol: the request itself is the primer, so prime at full depth, then poll TunableBarsCalculated() every 3s and hold the sweep until it stops changing (3 steady probes), then use whatever it settled at. Bounded at 10 min, and a give-up is logged as a give-up so an abandoned depth is never mistaken for a settled one. This supersedes 1dda479's clamp on the two training paths, which snapshotted a value that may still have been climbing; the clamp remains for the paths that cannot wait (inference/online/rescan/export, see 7e63a8b). The load-bearing part is what does NOT happen while waiting: no sweep. A 50k-bar feature scan starves the indicator threads the request just woke, which is how the failure sustained itself for 40 minutes at a time - discard era, re-sweep, discard, which is the 0->100% oscillation on the panel. Also adds IndicatorDepthReport(): per-handle BarsCalculated() on the priming, cap and stall lines. The logs proved WHICH FEATURE died but never WHICH HANDLE was short, so the cause had to be inferred - and was guessed wrong twice. The next occurrence reads it off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:44:27 -04:00
#define DEPTH_SETTLE_TIMEOUT_MS 600000
//--- ERA-BARRIER LIVENESS. How long a member may sit on the same era before the barrier stops
//--- treating it as one the others must wait for. See NoteBarrierProgress().
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
#define ENSEMBLE_BARRIER_STUCK_MS 720000
//--- HARD CAP on how far a member may run ahead of the SLOWEST still-training member, counting one
//--- excluded from the barrier. Stopping and naming the laggard beats running and producing
//--- nothing.
fix(ensemble): the era barrier read healthy startup work as a dead member Reported symptom: one member at era 17 while the rest sat at era 2, with the combined vote never scoring. Two faults compound to produce exactly that, and neither needs a broken model to trigger. FIRST - BUSY WAS READ AS STUCK. BarrierEraHeartbeat() decides liveness from one signal: has m_eraCount changed in the last 12 minutes. But Train() returns early, before the era loop, for three ONE-TIME phases that never touch m_eraCount - the label-cache prebuild, the pattern-DB backfill and the OOS simulation walk - and those are precisely what a slow topology spends its first many minutes doing. A member grinding steadily through a prebuild therefore looked identical to a dead one and was dropped from the barrier at startup, before it had trained a single era. The constant's own comment states the flawed premise: "comfortably past the slowest healthy ERA on the deepest chart" - true, and not the question being asked. Those three branches now call NoteBarrierProgress() and a chunk of phase work re-arms the watchdog exactly as an era does. SECOND - EXCLUSION HAD NO BOUND. Once dropped, a member is skipped by EnsembleMinTrainingEra(). Drop every OTHER member and that loop finds nothing to take a minimum over, falls through to its `return m_eraCount` fallback - the CALLER'S own era - and EnsembleEraBarrierHolds() evaluates `era > era`, false, for everybody. The barrier silently becomes a no-op and the fastest member runs away unbounded. EnsembleMinEraAnyMember() now measures against every still-training member, excluded or not, and a member may lead it by at most ENSEMBLE_MAX_ERA_LEAD eras. The cap is deliberately a real stop rather than a warning. A desynchronised ensemble is not a degraded one: the combined-vote score and the joint checkpoint both require every member on the same era, so weights trained past the cap can never be certified by any gate. The hold reports which of the two it is, because the operator's next move differs - an ordinary barrier hold resolves itself, a lead-cap hold names a member that needs diagnosing and will not resolve on its own. Not yet explained: "only one NN listened to the stop command". The panel now dispatches down the filter tree and reports the count it reached ("training stopped (N model(s))"), so the next run answers that definitively instead of leaving it to inference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 09:35:25 -04:00
#define ENSEMBLE_MAX_ERA_LEAD 4
//--- How often a held member says so in the journal. The panel line is written every call; this is the
//--- durable record, without which a frozen chart leaves no trace at all.
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
#define ENSEMBLE_BARRIER_REPORT_MS 120000
//--- Cadence of the settled per-era diagnostics when VerboseMode is off (see TrainLogDue): each repeating
//--- print fires on eras 0-3 and then every Nth. 25 is ~one block per 15 minutes per member at ~35s/era -
//--- enough to reconstruct a run without the 22MB/9.5h firehose. CHANGE events are never throttled.
feat(logs): throttle the settled per-era diagnostics - measured 22MB/9.5h of confirmed-working systems Measured from the journal (2026-08-19): the era deep-dive line (~2KB) plus the excursion verdict, tier re-rank, calibration move, barrier hold and selection-regressed note each printed EVERY era for EVERY member - ~940 eras/member/day - long after the systems they watch were confirmed working. Yesterday's file was 1.3GB (70% of it the news-filter calendar spam the sweep fix already removed). VerboseMode returns as an INPUT (demoted 2026-08-01 for the marketplace; that track is dead since the 2026-08-16 pivot) and gains a second job: false throttles each settled per-era print to eras 0-3 plus every TRAIN_LOG_EVERY_ERAS-th (25 ~= one deep-dive per ~15min per member); true restores the per-era firehose, flippable live. Never throttled: anything that marks a CHANGE - new bests, restores + eta decays, plateau stage transitions, deploy approvals, warnings, errors, the label-cache/adoption one-shots, and the combined-vote gate line (the active system's primary telemetry, still every era). Semantic fixes over blanket gating: - barrier hold now ARMS silently and prints only when the hold outlasts the 2-min report interval - a brief hold every era is the design, the long hold is the watchdog case the line exists for; - the ensemble deploy REFUSAL prints immediately when its reason changes (that is a finding), on cadence when unchanged; - the filtered-view census prints when its RESULT moves (drawn count, or strongest vote by >=2pp) and at least every 10th sweep. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 09:36:15 -04:00
#define TRAIN_LOG_EVERY_ERAS 25
//--- Gap between attempts to rebuild a dead indicator handle. Long enough not to hammer a terminal that
//--- is genuinely refusing, short enough to recover within one stall-report interval.
fix(indicators+panel): the dead handle is MEASURED now - recreate it; and order the ensemble panel by member, not by who published first THE ANSWER, off the instrumentation added in be39674, first run: ConvLSTM [HYB-2484]: TUNABLE INDICATOR REPORTS NO CALCULATED BARS - 1 tunable indicator(s) enabled and the least-ready answers BarsCalculated()=-1 ... Per-indicator depth: price=33982 MA=-1 ZigZag=33982 ATR=33982 MA=-1 with price, ZigZag and ATR all at full depth. **The handle is INVALID, not short.** Same line on USDJPY (price=50179 MA=-1). Depth was never the problem; the previous session's five theories were all answering the wrong question. And it is per-member, not per-chart: LSTM-2484 ran the 34-candidate auto-tune on that same chart at 15:24:18 and went on to train normally (feature health, 51 features, excursion head) reading the same indicator. Only ConvLSTM's handle - the last member constructed - was dead. WHY is still not established. All four members request ADMovingAverage with identical params, so MT5 hands them the SAME refcounted handle, and the tuner's inner loop is Create-then-IndicatorRelease over exactly that shared handle; that is the obvious suspect and it is NOT yet proven, so this commit does not act on it. 1. IndicatorDepthReport() NOW PRINTS HANDLE NUMBERS, not just depths. "MA=-1" says the handle is dead. "MA=-1(h12)" against another member's "MA=33982 (h12)" says it is the SAME handle and someone released it; "(h-1)" says it was never created. That is the difference between a refcount bug and a creation failure and it is one field. This is the measurement the shared-handle suspicion needs before anyone acts on it. 2. RECREATE A DEAD HANDLE INSTEAD OF SWEEPING AGAINST IT. A member that cannot read its own indicator must rebuild it. RepairDeadIndicatorHandles() re-Creates only the ENABLED tunables reporting BarsCalculated() < 0 - a merely COLD indicator (valid handle, 0 bars) is left alone to warm up the normal way. It does NOT release first: -1 means the terminal no longer knows the handle, so there is nothing to give back, and MT5 recycles handle VALUES so releasing a stale one could decrement whatever now owns that number. 30s cooldown, because every ServableBars() consumer reaches it including live inference on every tick. The feature cache is dropped with it, and the log names before/after depths. Cause-agnostic on purpose. Whatever is killing the handle, sweeping 50,163 bars against a buffer that answers EMPTY_VALUE at every index - then discarding the era and doing it again - is not a recovery. 3. THE SWEEP NOW HOLDS ON A DEAD HANDLE. ServableBars() keeps answering `want` (its contract; live inference and online learning have their own refusal paths and a 0 there reads as "no history at all"). SettledBars() - the training sweep's entry, the one caller that can afford to wait - returns 0 instead, so Train() holds and reports rather than burning a full-history pass it is guaranteed to throw away. A recreated handle is cold, so it primes through the existing settle path on the next call. If the repair fails the member holds indefinitely and says so every minute, and be39674's barrier liveness escape releases the rest of the ensemble after 12 minutes - which is the correct degradation and is exactly what the log shows happening. 4. THE PANEL ROWS WERE ORDERED BY WHO PUBLISHED FIRST. Reported on XAUUSD: LSTM, ConvLSTM, Perceptron, Convolutional instead of Perceptron, Convolutional, LSTM, ConvLSTM. ClaimEnsemblePanelSlot() handed out the next free row on each member's FIRST PublishStatus() call, so the order was a race - the members busy sweeping published before the ones sitting idle at the era barrier, and be39674 sharpened it by (correctly) making a held member stop writing the terse line. Rows are now keyed to m_ensembleIndex, the registration/construction order, which is fixed for the life of the chart. Claimed on every publish rather than once, so it is idempotent and refreshes the tag for a member whose ID was not final when it first published (the config-tag suffix is appended during InitIndicators, after EnsembleMember() registers). Unclaimed rows are skipped by the render and excluded from the model count, so a member that has not published yet leaves no gap and shifts nobody. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:40:00 -04:00
#define HANDLE_REPAIR_COOLDOWN_MS 30000
//+------------------------------------------------------------------+
//| sqrt(B) learning-rate compensation and the matching patience |
//| stretch. Both are exactly 1.0 at B=1, so the whole mini-batch |
//| apparatus vanishes when TRAIN_BATCH_SIZE is 1. |
//+------------------------------------------------------------------+
fix: the Adam second moment was never Adam - all four tiers Root cause of the B=32 regression, and it predates F4 entirely. Every Adam kernel stored v already square-rooted and then fed that stored value back in as if it were the variance: v_new = sqrt(b2 * v_old + (1 - b2) * g^2) That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below unit scale, so the denominator stops tracking the gradient and Adam degrades into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll (batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where a scale-invariant optimizer gives the same distance for both. After the fix all six magnitudes read 1.199 and v tracks |g| exactly. It hit conv/LSTM specifically because they sit behind a batch-norm with running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep in the degraded regime - while the dense stack near the loss stayed in the working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/ 0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already squared v back for gamma/beta and its comment named the kernels as wrong, which is exactly why gamma/beta kept training while the stages behind froze. Persisted .nnw needs no migration - v keeps its std-dev meaning. Also, the two ways F4 exposed it, both mine: - No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods (Krizhevsky 2014; Granziol et al. 2022), applied once in InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD. - Plateau patience denominated in eras, so raising B made the ladder 32x more impatient in its only unit. PAI converged at era 41 on ~49k updates where the same config had been finding new bests at era 1028. TrainPlateauPatienceEras() stretches it by the same sqrt(B). TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23 eras per stage, not 8 -> 45). Both helpers are identities at B=1. Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into the ranking key rather than checked at deploy time so a one-sided era cannot become best-so-far in the first place. Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded CAppDialog teardown that sat ahead of it - the same ordering inversion the rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit (vs ~1.1 s for the three that finished) having reached none of its cleanup, so its arrows stayed on the chart. Steps are now timed in the log. PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now walks every object type and reports the object counts when both are zero. Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt. FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
double TrainBatchLrScale(void) { return MathSqrt((double)TRAIN_BATCH_SIZE); }
int TrainPlateauPatienceEras(void) { return (int)MathRound(PLATEAU_PATIENCE_ERAS * MathSqrt((double)TRAIN_BATCH_SIZE)); }
//--- ONLINE CONTINUAL LEARNING (see OnlineLearnStep()). Live-chart-only: a deployed model keeps
//--- adapting to newly-RESOLVED bars on the same supervised triple-barrier task, never on trade
//--- P&L.
#define ONLINE_LEARN_MAX_CATCHUP 64
#define ONLINE_ACC_SMOOTH 50.0
#define ONLINE_LEARN_WARMUP 20
#define ONLINE_LEARN_MIN_ACC 40.0
#define ONLINE_LEARN_ACC_MARGIN 10.0
#define ONLINE_LEARN_PERSIST_EVERY 32
#define ONLINE_LEARN_MAX_CLASS_WEIGHT 5.0
//--- Pinned to the shipped defaults of the removed OversampleParity / ConstrainReplay / FocalLossGamma
//--- inputs - see the class-imbalance note above.
refactor(ai): nine class-imbalance inputs down to two The imbalance section offered nine controls for one job. Audited against the code, five of them did not do what their names said at the shipped defaults: AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns whenever the adjusted loss is on, which is default. OversampleParity DEAD in training - Training.mqh gated the replay loop on !useLogitAdjustedLoss (correctly, citing Buda et al. 2018). Live only in the online-learning path. EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma damper - "replay minority bars through pass-2 oversampling" was a focal-loss switch. ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25. UseStaticPrior An exact duplicate of FreezePriorCalibration - the two were OR'd together in the single place either is read. So they were not five mechanisms fighting; they were one mechanism plus eight knobs that mostly described machinery that no longer ran. That is worse than a real conflict, because the log agreed with the names: the label-cache line printed "reps up to 28x (90% parity) (seeding era 0's class-balance oversampling)" on every run, describing an oversampling pass that had been switched off. It is fixed here too - it cost this session a wrong diagnosis. The one genuine redundancy was focal loss, running at gamma*0.125 alongside the adjusted loss: two corrections on the same axis, the exact stacking failure this file already cited Buda et al. for in two other places, damped by a replay flag whose replay path was itself dead. Removed rather than re-tuned. The plateau ladder is unaffected - its escape is the learning-rate warm restart; the gamma anneal beside it only ever stepped toward zero. WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze: LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted- Loss boolean, since a strength dial where 0 already means off does not need an on/off switch beside it. FreezePriorCalibration unchanged. It is the only one of the six corrections with a consistency guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection already ranks on - so the loss and the deploy decision optimize one thing. The online continual-learning path keeps its own alpha-balanced focal weight, now as constants pinned to the removed inputs' shipped defaults, so its behaviour is unchanged. It legitimately needs its own correction: ApplyLogitAdjustment() only runs inside a training run, so a deployed model that was reloaded carries no logit offsets and would otherwise stream 31:1 data into itself uncorrected. The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a double fed to a %d conversion and had always emitted a literal 0; the |MR: segment is written as the constant its shipped defaults produced. Dropping either would have re-keyed every model and forced a from-scratch retrain of the one topology currently converged and trading. Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable "neutralized by prior correction" diagnostic. Both builds compile 0 errors, 0 warnings. No retrain forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
#define ONLINE_LEARN_PARITY 0.9
#define ONLINE_LEARN_ALPHA_CAP 3.0
#define ONLINE_LEARN_FOCAL_GAMMA 1.0
#define ONLINE_LEARN_ETA_SCALE 0.25
//+------------------------------------------------------------------+
//| Base learning rate for the selected optimizer. |
//| A free function, not a method: the constructor's init list needs |
//| it for both m_modelEta and m_etaCeiling, which runs before |
//| member-init order could safely let one depend on another. |
//| The sqrt(B) compensation is applied HERE, at the one point that |
//| decides the base rate, so it reaches the ceiling, the plateau |
//| boost and the anneal from a single edit. |
//+------------------------------------------------------------------+
double InitialEtaForOptimizer(void)
{
refactor(stdlib): adopt Math\Stat for the deploy gate's normal tail; retire the b1/b2/lr/momentum macros The gate's NormalUpperTail was a hand-rolled Abramowitz & Stegun 26.2.17 approximation. Its own comment gave the reason - "drags a chain of headers behind it" - and that turned out to be one file: Math\Stat\Normal.mqh includes only Math.mqh, which includes nothing. Swapped for Cody's rational approximation in the library (~18 significant digits vs |error| < 7.5e-8). No past verdict changes: at the z the gate operates on, the difference is orders of magnitude below DEPLOY_FAMILY_WISE_ALPHA. Adopting it needed the four bare macros in AI\Network.mqh gone first. "#define b1 AdamBeta1" collides with an identifier in Math.mqh, so the include would have macro-expanded the library's own local and failed to compile - the same landmine that made the original author rename the approximation's coefficients to ntB1..ntB5 rather than use the reference's b1..b5. lr, b2 and momentum are the same class of hazard: single-token global macros in a 52k-line codebase. All four now resolve to the input names they always aliased, which is a pure textual identity - verified zero bare occurrences remain. Also: - SelectionSort over the buffered signals was O(n^2) with an O(n^2) count of StructToTime calls, because the comparison rebuilt both datetimes from the six int date fields every time. Now materialises the keys once and does an insertion sort; ArraySort cannot permute a struct array. IsEarlier goes with it, MakeDateTime becomes SignalTime. - Seven FileOpen sites lacked FILE_SHARE_READ|FILE_SHARE_WRITE, including AtomicWriteBegin, which stages every model save. All 43 sites now carry them - an exclusive open fails outright when another process holds the path, which here has meant a silently skipped save. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 19:31:36 -04:00
return ((TrainingOptimizer == SGD) ? SgdLearningRate : AdamLearningRate) * TrainBatchLrScale();
}
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//+------------------------------------------------------------------+
//| Uniform random index in [0, n) for Fisher-Yates shuffles. |
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//+------------------------------------------------------------------+
int ShuffleRandomIndex(const int n)
{
feat(rng): ALGLIB's L'Ecuyer generator replaces MathRand, and a seed collision goes with it MQL5's MathRand() is the 15-bit MSVC LCG - 32768 distinct values and the lattice structure that shape of generator has. Two places here actually lean on randomness and both were hurt by it: WEIGHT INIT. Six He/LeCun-uniform sites drew ((MathRand()+1)/32768.0 - 0.5) * 2 * scale, so a first dense layer of ~250k weights had only 32768 possible values and thousands of connections started byte-identical. Breaking that symmetry is the whole job of random init. SHUFFLING. ShuffleRandomIndex() already had to splice TWO MathRand() draws to reach 30 bits, and its own comment documented the residual modulo bias it still carried. HQRndUniformI() is rejection-sampled and exactly uniform, so the splice and the bias note both go. CHighQualityRand is L'Ecuyer's combined multiplicative congruential generator - two differenced streams, 31-bit output, period ~2.3e18 - and it ships with the terminal. AND A BUG THE MIGRATION EXPOSED. The three MathSrand(GetTickCount()) calls sit immediately before "build a fresh topology", once per model. GetTickCount() steps in ~15.6 ms on Windows and an ensemble builds every member inside one OnInit, so members could be handed the SAME seed and draw the SAME weights wherever their shapes coincide - and members that start identical are not an ensemble. WarriorRandSeed() takes a salt (the model id) plus a never-reset call counter, so a collision is impossible rather than merely unlikely, while the tick keeps the run itself genuinely unrepeatable the way those call sites asked for. Seeds are masked positive rather than trusted: HQRndSeed computes s % (M-1) + 1 and MQL5's % keeps the sign, so a negative seed leaves the generator in a state its own assertions reject. GetTickCount() is a uint and goes negative as an int after ~24 days of uptime - a fault that would surface as "training is broken" on a long-running terminal and nowhere else. The indicator tuner's 52 draws move across too: its random search is where sample quality earns its keep. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 00:29:12 -04:00
return WarriorRandInt(n);
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
}
class CExpertSignalAIBase : public CExpertSignalCustom
{
protected:
string ID;
//+------------------------------------------------------------------+
//| ID with the bracketed config tag stripped: "Hybrid 3L [HYB-9369]" |
//| -> "Hybrid 3L". The tag tells one CHART's model files from |
//| another's, which is a developer's problem, not an owner's. Logs |
//| and the verbose panels keep the full ID. Strips from the LAST |
//| " [" so a model name containing a bracket cannot truncate more |
//| than intended. |
//+------------------------------------------------------------------+
string DisplayName(void) const
{
int cut = StringFind(ID, " [");
int next = cut;
while(next >= 0)
{
cut = next;
next = StringFind(ID, " [", cut + 1);
}
return (cut >= 0 ? StringSubstr(ID, 0, cut) : ID);
}
//--- Per-MEMBER arrow namespace: "WarSig_PAI_", "WarSig_CONV_", ... Global purges still match on
//--- bare "WarSig_".
fix(ensemble): per-member arrow namespaces; ConvLSTM rename; dialog in purge list The ensemble chart UI had a shared-namespace defect that answered the user question "what do the arrows represent?" with "a bug": all four members drew arrows under the same WarSig_<bartime> object names, so the chart showed whichever member rendered LAST, one member Neutral deleted another member Buy at the same bar, each member init sweep wiped the arrows the previous member had just restored, and SaveChartSignals - which rebuilds the sidecar by SCANNING the chart - persisted every other member arrows into its own history (the exact cross-model laundering its own header warns about, now happening BETWEEN ensemble members). Arrows are now namespaced per member (WarSig_PAI_, WarSig_CONV_, WarSig_LSTM_, WarSig_HYB_): draw, delete, restore, prune, member init sweep, destructor purge and the sidecar scan are all member-scoped, and the tooltip names the model. Global purges keep matching the bare WarSig_ prefix, which covers all member namespaces plus old-format leftovers from earlier builds. Labels: the ensemble panel header no longer says "HYBRID ensemble" (HYBRID is one member; the header is the ensemble) and the CONVLSTM member displays as ConvLSTM instead of Hybrid. Its SHORT id stays HYB deliberately - it names the model folder and changing it would orphan every model trained under that path. Deinit: the alt-data mapping dialog namespace (WarriorAltMap_) joins WarriorChartPrefixes, so both the OnInit purge and the deinit final sweep now cover it - it was in neither list, so a dialog starved of its own Destroy() left its controls on the chart permanently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 18:26:55 -04:00
string ArrowPrefix(void) const { return SIG_ARROW_PREFIX + m_id + "_"; }
//--- Every subclass of this one is a neural net. Tells the aggregate's raw-arrow layer that this
//--- filter draws its OWN arrows, from cached per-bar scans spanning the whole chart, and must not be
//--- drawn again from the once-per-bar live path.
feat(chart): filtered view - one arrow per trade the bot would actually take Adds DrawUnfilteredSignals (default OFF) and, with it off, replaces the per-model arrow layer with the decision the EA would really have made. THE FILTERED ARROW IS DRAWN AT THE ORDER, NOT AT THE THRESHOLD. Clearing Min_Vote_Open is not the same as trading: a setup can pass the vote and still never reach the broker (invalid SL/TP, stops-level, ATR warm-up, unsynced swing history), and every one of those lands in OpenParams' failure branch. So DrawVoteArrow() fires only after the order parameters validate, and the failure branch withdraws any arrow already standing on that bar. One arrow is one entry the EA would have placed - carrying the vote, the threshold it cleared, and the SL/TP the order would have had. Classic signals now draw too, under their own name and weight, so a chart running MA/RSI/MACD/Ichimoku alongside the nets reads the same way an ensemble chart does. They can only be drawn from the aggregate's once-per-bar pass, because unlike the AI members they have no cached per-bar scan. Two subtleties that would each have produced a quietly wrong chart: - The raw classic draw sits AFTER filter.Direction(), not beside the journaling block. GetActivePattern*() are CONSUMING reads holding the PREVIOUS evaluation - "one tick later", which at Expert_EveryTick=false is one BAR later. Keyed off those and placed at StartIndex(), every classic arrow would have been drawn one bar early, which on a chart is indistinguishable from a model that genuinely leads. Peek*() accessors (non-consuming) let pattern, weight and bar come from one evaluation. - CExpertSignalAIBase::DrawObject() early-returns instead of gating its five call sites, so the switch cannot be honoured in three passes and missed in the fourth. Its delete counterparts stay ungated so flipping the input off and rescanning clears the raw layer rather than stranding it. SIG_ARROW_PREFIX and g_signalsVisible move from ExpertSignalAIBase.mqh down to ExpertSignalCustom.mqh - the nearest common ancestor - because the classic signals cannot see the AI header (it is included later in Warrior_EA.mq5). The vote layer gets SIG_VOTE_PREFIX under the same bare prefix, so WarriorChartPrefixes()' purge still reaches every arrow without knowing they exist. NOT YET BUILT: the reconstructed history behind attach. Filtered arrows currently start where the EA starts. See the next commit. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:41:28 -04:00
virtual bool IsAIFilter(void) const override { return true; }
CiOpen m_Open;
CiClose m_Close;
CiHigh m_High;
CiLow m_Low;
CiVolumes m_Volumes;
CiTime m_Time;
//--- Optional classic-indicator input FEATURES, independent of the CSignal* instances used for
//--- voting: feature engineering and signal voting are unrelated hierarchies. Periods come from the
//--- tuner. Built-in Ci* wrappers throughout, so none of them carries a CiCustom depth limit.
feat(indicators): run the built-in iMA and MetaTrader's ZigZag; add a classic-vote shift MA: CustomIndicators\ADMovingAverage is replaced by the built-in iMA (CiMA) on both consumers - the classic vote and the NN MA input feature. This drops the five advanced types ALMA/DEMA/ZLEMA/T3/Kalman, which have no iMA equivalent; MA_TYPE_PRESETS is now ENUM_MA_METHOD's own codes and the tuner searches all four. It also removes a documented failure mode: a custom indicator's depth is bounded by TERMINAL_MAXBARS, and m_MA was the one whose feature block REJECTS the bar on a short read - the "feature 25 fails on every bar" incident of 2026-08-17. A built-in is served at any depth. MIGRATION. SMA moves from code 5 to 0, so persisted type codes change meaning. SanitizeMaType() is the single validity rule; TunedPeriods records now carry a version field and a v1 record remaps 5..8 -> 0..3, falling back to SMA for a stored advanced type (unrecoverable - old 0..4 are indistinguishable from valid new codes). Existing .nnw files re-key on their own, because MA_Type is hashed into the topology fingerprint, so models retrain rather than silently running on different MA values. EXPECT A FULL RETRAIN. ZigZag: ADZigZag was a byte-identical rename of MetaQuotes' Examples\ZigZag - verified by normalising identifiers and stripping comments, 233 significant lines each with only renamed symbols differing. It now loads the stock one, so nothing is bundled and MetaQuotes' fixes arrive without a rebuild here. Both #resource entries are gone. Classic_Shift: a new input, the BAR the four classic votes evaluate on (0 = forming, 1 = last closed, default 1). One implementation on CExpertSignalCustom, inherited by all four rather than repeated per module. Defaults to a sentinel meaning "unset", so the AI signals and the aggregate keep the stock every_tick rule and their feature/label alignment is untouched. The META corpus sweep still takes precedence. CExpertBase::StartIndex turns out to be virtual, so this is a real override, not the name-hiding the old comment claimed. Not compiled - MetaEditor compile pending. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 19:09:58 -04:00
CiMA m_MA;
CiRSI m_RSI;
CiMACD m_MACDFeature;
CiIchimoku m_Ichimoku;
//--- Custom price-action/volume indicators (CustomIndicators\*.mq5), loaded via iCustom/CiCustom.
CiCustom m_ADCumulativeDelta;
CiCustom m_ADShorteningOfThrust;
CiCustom m_ADWyckoffEventStream;
CiCustom m_ADWyckoffFailedStructure;
CiCustom m_ADWyckoffSignificantBarInversion;
//--- "Is this AD indicator still calculating?" - cold must be a TRANSIENT rejection, never a
//--- zero-fill; see the definition in Features.mqh.
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
bool ADIndicatorCold(CiCustom &ind, string block);
//--- Stamp of the last pass-1 sweep in which EVERY window failed on a transient cause. Non-zero arms
//--- a short era-start backoff so the retry loop stops starving the indicator threads it waits on.
2026-08-13 10:23:11 -04:00
uint m_coldSweepTick;
//--- Last depth ServableBars() had to clamp to. Held so the explanation prints when the cap CHANGES
//--- rather than once per era per call site. 0 = never clamped.
fix(train): clamp the sweep to indicator-servable depth - the scan wall was CopyBuffer, not a cold indicator Symptom: on a 3-chart run with contention ruled out (SP500 sitting at era 2552), USDJPY and XAUUSD produced 0 usable windows out of 50,163 and 33,966 - forever, re-sweeping on every discard, which is the panel oscillating 0->100%. Bars() is the PRICE series depth. A CUSTOM indicator's is not: MT5 calculates it in its own context bounded by "Max bars in chart" (TERMINAL_MAXBARS), and CopyBuffer past that limit does not short-read, it FAILS - so CDoubleBuffer keeps nothing and EVERY index answers EMPTY_VALUE. ADMovingAverage is the only custom indicator whose feature block REJECTS on EMPTY_VALUE (ADZigZag, also CiCustom, neutral-fills; RSI/MACD/Ichimoku/ATR are built-ins served at any depth), so the sweep died on feature 25 of every bar while the 24 price features under it were fine. That is exactly the "window had 24 of 832 values" the stall report named. Perfectly depth-correlated, measured 2026-08-17: SP500 16,234 bars -> era 2552 XAUUSD 33,982 -> 0 windows XTIUSD 16,611 bars -> era 71 USDJPY 50,179 -> 0 windows This RETIRES the 2026-08-17 cold-indicator reading of the same stall. f0cf659 was right that the rejection must be transient and that the dead backoff had to arm - the branch did change to 'cold-indicator backoff' - but waiting cannot fix a depth the terminal will never grant. So Train() now clamps to TunableBarsCalculated() (which existed and was only ever used for a tuner printout) and trains on the history that IS available, naming TERMINAL_MAXBARS in the log so the cause is readable next time. m_coldSweepTick still owns the genuinely transient case: that reads back as -1, not a positive short count. Recomputed per era, so the clamp lifts by itself if the setting is raised. Also fixes a real off-by-one it was hiding: the MA block reads GetData(idx) AND GetData(idx + 1) for its bar-over-bar change, but ResizeBuffers sized m_MA to barIndex exactly - so the deepest bar of every sweep read one past the end and was rejected as cold. Same shape as the +ichiKijun the Ichimoku/close pair already has. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:27:14 -04:00
int m_indicatorDepthCapBars;
//--- One-shot latch for "an enabled tunable indicator reports NO calculated bars". Distinct from the
//--- cap latch: that means "serves less than asked" and is recoverable, this means a dead handle with
//--- no depth to clamp to. Cleared when a real depth returns, so a second outage is still reported.
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
bool m_indicatorDepthDeadWarned;
//--- Cooldown between attempts to rebuild a dead handle. Every ServableBars() consumer can reach the
//--- repair - training, live inference, online learning - so a refusing terminal must not be hammered.
fix(indicators+panel): the dead handle is MEASURED now - recreate it; and order the ensemble panel by member, not by who published first THE ANSWER, off the instrumentation added in be39674, first run: ConvLSTM [HYB-2484]: TUNABLE INDICATOR REPORTS NO CALCULATED BARS - 1 tunable indicator(s) enabled and the least-ready answers BarsCalculated()=-1 ... Per-indicator depth: price=33982 MA=-1 ZigZag=33982 ATR=33982 MA=-1 with price, ZigZag and ATR all at full depth. **The handle is INVALID, not short.** Same line on USDJPY (price=50179 MA=-1). Depth was never the problem; the previous session's five theories were all answering the wrong question. And it is per-member, not per-chart: LSTM-2484 ran the 34-candidate auto-tune on that same chart at 15:24:18 and went on to train normally (feature health, 51 features, excursion head) reading the same indicator. Only ConvLSTM's handle - the last member constructed - was dead. WHY is still not established. All four members request ADMovingAverage with identical params, so MT5 hands them the SAME refcounted handle, and the tuner's inner loop is Create-then-IndicatorRelease over exactly that shared handle; that is the obvious suspect and it is NOT yet proven, so this commit does not act on it. 1. IndicatorDepthReport() NOW PRINTS HANDLE NUMBERS, not just depths. "MA=-1" says the handle is dead. "MA=-1(h12)" against another member's "MA=33982 (h12)" says it is the SAME handle and someone released it; "(h-1)" says it was never created. That is the difference between a refcount bug and a creation failure and it is one field. This is the measurement the shared-handle suspicion needs before anyone acts on it. 2. RECREATE A DEAD HANDLE INSTEAD OF SWEEPING AGAINST IT. A member that cannot read its own indicator must rebuild it. RepairDeadIndicatorHandles() re-Creates only the ENABLED tunables reporting BarsCalculated() < 0 - a merely COLD indicator (valid handle, 0 bars) is left alone to warm up the normal way. It does NOT release first: -1 means the terminal no longer knows the handle, so there is nothing to give back, and MT5 recycles handle VALUES so releasing a stale one could decrement whatever now owns that number. 30s cooldown, because every ServableBars() consumer reaches it including live inference on every tick. The feature cache is dropped with it, and the log names before/after depths. Cause-agnostic on purpose. Whatever is killing the handle, sweeping 50,163 bars against a buffer that answers EMPTY_VALUE at every index - then discarding the era and doing it again - is not a recovery. 3. THE SWEEP NOW HOLDS ON A DEAD HANDLE. ServableBars() keeps answering `want` (its contract; live inference and online learning have their own refusal paths and a 0 there reads as "no history at all"). SettledBars() - the training sweep's entry, the one caller that can afford to wait - returns 0 instead, so Train() holds and reports rather than burning a full-history pass it is guaranteed to throw away. A recreated handle is cold, so it primes through the existing settle path on the next call. If the repair fails the member holds indefinitely and says so every minute, and be39674's barrier liveness escape releases the rest of the ensemble after 12 minutes - which is the correct degradation and is exactly what the log shows happening. 4. THE PANEL ROWS WERE ORDERED BY WHO PUBLISHED FIRST. Reported on XAUUSD: LSTM, ConvLSTM, Perceptron, Convolutional instead of Perceptron, Convolutional, LSTM, ConvLSTM. ClaimEnsemblePanelSlot() handed out the next free row on each member's FIRST PublishStatus() call, so the order was a race - the members busy sweeping published before the ones sitting idle at the era barrier, and be39674 sharpened it by (correctly) making a held member stop writing the terse line. Rows are now keyed to m_ensembleIndex, the registration/construction order, which is fixed for the life of the chart. Claimed on every publish rather than once, so it is idempotent and refreshes the tag for a member whose ID was not final when it first published (the config-tag suffix is appended during InitIndicators, after EnsembleMember() registers). Unclaimed rows are skipped by the render and excluded from the model count, so a member that has not published yet leaves no gap and shifts nobody. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:40:00 -04:00
uint m_handleRepairTick;
//--- ERA-BARRIER LIVENESS STATE (see ENSEMBLE_BARRIER_STUCK_MS and BarrierEraHeartbeat()).
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
long m_barrierEraSeen;
uint m_barrierEraTick;
//--- Set by NoteBarrierProgress() when a long one-time phase advances a chunk, so the watchdog can
//--- tell "busy" from "stuck" - the distinction it could not make before.
fix(ensemble): the era barrier read healthy startup work as a dead member Reported symptom: one member at era 17 while the rest sat at era 2, with the combined vote never scoring. Two faults compound to produce exactly that, and neither needs a broken model to trigger. FIRST - BUSY WAS READ AS STUCK. BarrierEraHeartbeat() decides liveness from one signal: has m_eraCount changed in the last 12 minutes. But Train() returns early, before the era loop, for three ONE-TIME phases that never touch m_eraCount - the label-cache prebuild, the pattern-DB backfill and the OOS simulation walk - and those are precisely what a slow topology spends its first many minutes doing. A member grinding steadily through a prebuild therefore looked identical to a dead one and was dropped from the barrier at startup, before it had trained a single era. The constant's own comment states the flawed premise: "comfortably past the slowest healthy ERA on the deepest chart" - true, and not the question being asked. Those three branches now call NoteBarrierProgress() and a chunk of phase work re-arms the watchdog exactly as an era does. SECOND - EXCLUSION HAD NO BOUND. Once dropped, a member is skipped by EnsembleMinTrainingEra(). Drop every OTHER member and that loop finds nothing to take a minimum over, falls through to its `return m_eraCount` fallback - the CALLER'S own era - and EnsembleEraBarrierHolds() evaluates `era > era`, false, for everybody. The barrier silently becomes a no-op and the fastest member runs away unbounded. EnsembleMinEraAnyMember() now measures against every still-training member, excluded or not, and a member may lead it by at most ENSEMBLE_MAX_ERA_LEAD eras. The cap is deliberately a real stop rather than a warning. A desynchronised ensemble is not a degraded one: the combined-vote score and the joint checkpoint both require every member on the same era, so weights trained past the cap can never be certified by any gate. The hold reports which of the two it is, because the operator's next move differs - an ordinary barrier hold resolves itself, a lead-cap hold names a member that needs diagnosing and will not resolve on its own. Not yet explained: "only one NN listened to the stop command". The panel now dispatches down the filter tree and reports the count it reached ("training stopped (N model(s))"), so the next run answers that definitively instead of leaving it to inference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 09:35:25 -04:00
bool m_barrierPhaseProgress;
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
bool m_barrierExcluded;
uint m_barrierHoldReportTick;
//--- One-shot latch for the live-inference hold in RefreshConvergedSignal(). Cleared when the depth
//--- returns, so a second outage is reported rather than swallowed.
fix(depth): route EVERY ResizeBuffers call site through one indicator-depth gate 1dda479 clamped the training sweep. It left five other paths asking the indicators for a depth they cannot serve, and on a live account the quiet ones are worse than the stall was - a stalled chart is visible, a chart trading on a degraded feature window is not. ServableBars(want, context) is now the single gate, and all six go through it: training sweep clamp, floored at TRAIN_MIN_CLAMPED_BARS (below that a small positive BarsCalculated is warm-up, which m_coldSweepTick owns) label prebuild clamp - labels come from price/ADZigZag and would survive a capped MA, but ResizeBuffers sizes EVERY buffer and a failed CopyBuffer leaves m_MA EMPTY for the next reader, so this path could silently re-break the block Train()'s clamp just fixed live inference HOLD. Below `need` the swing block takes its degraded path and inference runs on a different feature distribution than the model was fitted on. This EA sizes real positions off that output, so no signal beats a mismatched one online learning HOLD, same reason and worse - this path WRITES to a live trading model, so a mismatched (features, label) pair is not a wrong arrow, it is a wrong weight update that compounds every bar chart rescan clamp - SIGNAL_RESCAN_LOOKBACK_BARS is 5000 and MT5's smallest "Max bars in chart" is also 5000, so this one is genuinely reachable; uncapped it repaints the window all-Neutral research export clamp before the emptiness test, so a capped symbol exports the depth it has rather than writing a CSV with a dead feature block - an artefact that looks complete and is silently wrong Both HOLDs are insurance, not expected states: `need` tops out near 1,152 bars (16 + 750 + 384 + 2) against a 5,000 floor on the terminal setting. They exist so the failure mode is unreachable rather than merely unlikely. Not changed: a genuinely SHORT price history still takes the old degraded path at every site. That is pre-existing behaviour and narrowing it would mute charts that trade today, so it stays a separate decision rather than a side effect of this fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:33:16 -04:00
bool m_inferenceDepthRefusalWarned;
//--- One-shot latch for the label-prebuild block message: a prebuild that cannot prepare its buffers
//--- retries on every scheduled call forever.
fix(buffers): revert the MA +1 - it asked for a bar that does not exist and stopped every chart REGRESSION I INTRODUCED IN 1dda479, live for ~15 minutes. CSeries::BufferResize -> CheckLoadHistory -> CheckTerminalHistory succeeds only when Bars() >= size. Train() calls ResizeBuffers with barIndex == Bars(), so sizing the MA buffer to barIndex + 1 asks for one bar more than the symbol has and fails the WHOLE ResizeBuffers call. The log named it exactly: failed to get 50180 bars for USDJPY,PERIOD_H4 (Bars() = 50,179) failed to get 33983 bars for XAUUSD,PERIOD_H4 (Bars() = 33,982) StartLabelCachePrebuild() then bailed on the false return and stayed silent, so the only symptom was Train() reporting "arming the first label-cache prebuild" forever with labelCacheBars=0 - the panel's "getting ready". The premise was wrong, not just the arithmetic. The MA block reads GetData(idx) AND GetData(idx + 1), and at the OLDEST bar that second read is SUPPOSED to fail - there is no older bar to difference against. Rejecting that one bar is correct behaviour; buying it cost the entire history. Two more things, since the same defect had a second instance and no alarm: - The Ichimoku pair (closeBars and m_Ichimoku, both barIndex + ichiKijun) is the same bug with a far larger constant, latent only because the feature is off. Both are now clamped to Bars(). The oldest ichiKijun bars then have no cloud, which that block's EMPTY_VALUE guard already handles per-bar - the right outcome. - The prebuild's bare `return` on a false ResizeBuffers now says so once, naming the depth and Bars(). MQL5's own "failed to get N bars" was in the log the entire time, from a stack frame nothing connected to the prebuild. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 14:12:47 -04:00
bool m_prebuildBlockWarned;
//--- SettledBars() probe state. m_depthSettleStart doubles as the "a wait is in progress" flag.
feat(depth): prime -> settle -> sweep, and name which handle is short "Max bars in chart" is set to Unlimited, so the static-terminal-limit reading in 1dda479 was wrong. Two other candidate causes are falsified too: the price series is fully downloaded and flat (USDJPY 50,162 -> 50,163 over 80 minutes, i.e. one new H4 bar), and the handles have been stable since 13:07 with zero windows for the 17 minutes after, so it is not download-in-progress and not handle churn. The MA period tops out at 200 (ADIndicatorTuner MA_PERIOD_PRESETS) against ~50k bars, so it is not indicator cost either. What IS verified stays verified: CopyBuffer past the calculated depth fails outright rather than short-reading, so the buffer holds nothing and every index reads EMPTY_VALUE; m_MA is the only CiCustom whose block REJECTS on that (m_ADZigZag neutral-fills, RSI/MACD/Ichimoku/ATR are built-ins); the wall is therefore feature 25 of every bar, exactly as the "24 of 832" stall lines said. And it is depth-correlated: 16k-bar charts train, 34k/50k get zero windows forever. So the WHY is still open, and this fix does not depend on it. Per the user's protocol: the request itself is the primer, so prime at full depth, then poll TunableBarsCalculated() every 3s and hold the sweep until it stops changing (3 steady probes), then use whatever it settled at. Bounded at 10 min, and a give-up is logged as a give-up so an abandoned depth is never mistaken for a settled one. This supersedes 1dda479's clamp on the two training paths, which snapshotted a value that may still have been climbing; the clamp remains for the paths that cannot wait (inference/online/rescan/export, see 7e63a8b). The load-bearing part is what does NOT happen while waiting: no sweep. A 50k-bar feature scan starves the indicator threads the request just woke, which is how the failure sustained itself for 40 minutes at a time - discard era, re-sweep, discard, which is the 0->100% oscillation on the panel. Also adds IndicatorDepthReport(): per-handle BarsCalculated() on the priming, cap and stall lines. The logs proved WHICH FEATURE died but never WHICH HANDLE was short, so the cause had to be inferred - and was guessed wrong twice. The next occurrence reads it off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:44:27 -04:00
uint m_depthSettleStart;
uint m_depthProbeTick;
int m_depthProbeLast;
int m_depthProbeStable;
//--- Ground truth for the training labels - MetaTrader's own Examples\ZigZag. Always created, never
//--- gated behind an Enable* input because it is not an optional feature, it IS the label; and never
//--- touched by AutoTuneIndicators, because tuning the ground truth alongside the model scored
//--- against it would let a trial "improve" by cherry-picking an easier target.
CiCustom m_ADZigZag;
//--- Live tunable values for each AD indicator plus their flatten/perturb/best-tracking logic.
CADIndicatorTuner m_indicatorTuner;
bool m_autoTuneIndicators;
//--- Rebuilds only the AD* handles in place, so ReInit picks up updated param structs.
bool ReInitADIndicators(CIndicators *indicators);
//--- Installs a param set into the tuner and rebuilds handles ONLY when the set actually differs from
//--- what the indicators already run - see the definition for the resume-time churn this avoids.
2026-08-13 10:23:11 -04:00
bool AdoptIndicatorParams(const double &loaded[], CIndicators *indicators);
//--- Builds a fresh untrained topology into Net. Split from InitNeuralNetwork() so the tuner can
//--- rebuild weights per trial without re-running indicator init, which would Add() them twice.
bool BuildFreshTopology();
//--- Retained so TuneIndicatorsAndTrain() can call ReInitADIndicators() between trials.
CIndicators *m_indicatorsPtr;
CNet *Net;
//--- EMA "shadow" copy of Net, blended a SHADOW_WEIGHT_TAU step toward Net at the end of every
//--- era rather than replaced. Live inference reads THIS, so any single era's raw weights -
//--- including an Adam overshoot - can only nudge what is deployed, never overwrite it.
CNet *m_shadowNet;
//--- One-shot latch for the clone bootstrap. Cloning a second net can fail on the tester's CPU-
//--- DLL fallback, and without this the retry would re-initialise the compute backend on EVERY
//--- bar.
bool m_shadowBootstrapAttempted;
//--- ONLINE CONTINUAL-LEARNING STATE (see OnlineLearnStep(); tunables at ONLINE_LEARN_*). The
//--- watermark is a bar TIME, not a now-relative index, so it survives the per-bar index-frame
//--- shift.
bool m_enableOnlineLearning;
datetime m_onlineLearnedUpToTime;
double m_onlineRollingAcc;
long m_onlineSamples;
int m_onlineBarsSincePersist;
//--- Latched log state so the guardrail freeze/resume transition prints once per flip, not per bar.
bool m_onlineBlendFrozen;
CArrayDouble *TempData;
double dError;
double dUndefine;
double dForecast;
double dPrevSignal;
//--- ALTERNATION GATE REMOVED 2026-08-01 with the triple-barrier relabel. m_lastNonNeutralSignal
//--- suppressed any live Buy following another Buy with no Sell between. Do not reinstate it. It
//--- also meant a one-sided (`Sell:0%`) model got ONE trade per backtest, because the awaited
//--- opposite signal that reopens the gate never came.
diag: inference-path census, to explain zero-trade backtests A backtest of the CONVERGED CONV model produced "Final directional result: 0.00000000" on every one of 1744 bars and therefore zero trades. Nothing in the log could separate the three candidate causes, and each needs a different fix: 1. RefreshLatestSignal never called (new-bar gate never fires) 2. called, but bailing at one of its two early returns 3. running fine, and the model genuinely answers Neutral every bar Counts all three plus the Buy/Sell/Neutral split, printed once at shutdown via StopTraining (which the tester reaches through OnDeinit). Three increments per bar against a full feedForward - not worth gating. Ruled out while writing this, so the next session does not re-derive it: - the alternation gate (m_lastNonNeutralSignal) is NOT the cause. It starts at Neutral, so a first Buy would still fire and show up as one non-zero direction. We saw zero. It IS still a live hazard for a one-sided model - CONV currently calls Buy:17% Sell:0%, and after the first Buy every later Buy is suppressed until a Sell that never comes - but it cannot explain an all-zero run. - shallow buffers do not hard-fail the feature builder: the swing-context Donchian loop breaks gracefully when it runs off loaded history. It does mean converged-path inference computes Donchian/return/SMA features over a TRUNCATED window versus training, which is a real train/inference skew worth its own fix, but it degrades features rather than zeroing them. Both builds 0/0. Diagnostic only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 18:24:32 -04:00
long m_refreshOk;
long m_refreshFailFeatures;
long m_refreshFailShort;
long m_refreshBuy;
long m_refreshSell;
long m_refreshNeutral;
//--- VOTE-GATE census. Without these two the census reads as "the model answers Neutral" -
//--- false, and it points at a completely different fix. They separate the model's ANSWER from
//--- whether that answer was allowed to become a vote.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
long m_voteGateBlocked; // directional decisions the readiness gate discarded
long m_voteGatePassed; // directional decisions that became a real vote
//--- Flag pair as of the first vote attempt, latched so the tally can name WHICH half of the gate
//--- failed rather than just reporting that it did. -1 = no vote was ever attempted.
int m_voteGateCompleteAtFirst;
int m_voteGateLoadedAtFirst;
//--- Non-max suppression window for directional signals: keeps the FIRST bar of a same-direction
//--- run and drops same-direction neighbours within it, 0 disables. Per-direction, so a missed
//--- opposite signal never blocks a later reversal, and causal, so live and drawn history
//--- declutter alike.
int m_signalClusterWindow;
//--- Per-bar predicted signed signal for THIS era (index = now-relative bar index; -2 = not
//--- scored this era). Recording (not drawing) also decouples NMS from pass 2's SHUFFLED order,
//--- which no inline cursor could dedup.
double m_arrowSignalCache[];
//--- Live-side NMS state: bar TIME of the last SEEN signal per direction (advances on every same-
//--- direction bar, kept or suppressed, so a contiguous live run collapses to one) plus the cached
//--- accept/suppress decision for that exact bar (keeps repeated same-bar RefreshLatestSignal calls
//--- idempotent - re-evaluating the same bar returns its first decision, not a flipped one). 0 = none.
datetime m_nmsLiveBuyTime;
datetime m_nmsLiveSellTime;
bool m_nmsLiveBuyAccept;
bool m_nmsLiveSellAccept;
//--- Last KEPT live signal of either direction, for cross-direction resolution: a Buy and a Sell
//--- within m_signalClusterWindow bars are flicker at one turn zone (real opposite pivots are a
//--- whole leg apart), so only the higher-confidence side is kept.
datetime m_nmsLiveKeptTime;
ENUM_SIGNAL m_nmsLiveKeptDir;
double m_nmsLiveKeptConf;
datetime dtStudied;
long m_eraCount; // cumulative era counter, persisted in the .nnw so restarts don't look like they reset progress
bool m_trainingComplete; // persisted: true only once Train() converged (objective+stability), not just interrupted
bool bEventStudy;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
//--- This instance's study-event id (STUDY_EVENT_ID_BASE + construction order) and the tick-count
//--- when bEventStudy was last armed - see the STUDY_EVENT_ID_BASE comment for why these exist.
//--- All arming goes through ArmStudyEvent() so the id and the watchdog stamp can never drift apart.
ushort m_studyEventId;
uint m_studyArmedTick;
//--- out-of-sample holdout: share (%) of the study period never trained on, used only to
//--- measure genuine forward accuracy so overfitting shows up in the stats, not just live/OOS trading
int m_oosSplitPct;
double dOosError; // smoothed OOS mismatch rate (0..100), lower is better
double dOosForecast; // smoothed OOS accuracy (0..100)
int m_oosSamples; // count of OOS predictions evaluated this Train() call
//--- per-era raw (pre-softmax) output-neuron stats over pass 3's OOS scan, reset at pass 3
//--- start; surfaced in the era-end log line.
double m_oosOutMin[3];
double m_oosOutMax[3];
double m_oosOutSpreadSum;
int m_oosOutCount;
//--- WHY "Neutral" WON, per OOS bar. ApplyClassificationSoftmax() requires a STRICT majority and
//--- sends every tie to Neutral, so one label covers two events needing OPPOSITE fixes: strict -
//--- the net really ranks Neutral highest.
feat(diagnostics): split a reported "Neutral" into CHOSE vs TIED - they need opposite fixes ApplyClassificationSoftmax() requires a STRICT majority over both rivals and sends every tie, 2-way or 3-way, to Neutral. So "OOS recall Neutral:100%" is two completely different events sharing one label: CHOSE - the net genuinely ranks Neutral highest. A class-prior/label problem. TIED - the top two are EXACTLY equal, so the net expressed no preference and the tie-break reported Neutral. A SATURATION problem: the head is SIGMOID, and a saturated sigmoid returns exactly 0.0f or 1.0f in the DLL's float32, so two classes pinned to the same rail compare equal and the bar is silently discarded. Nothing in the logs could tell them apart, and the fixes point opposite ways. Eras 1-25 of the 2026-08-17 solo PAI run read "Neutral 100%" at spread avg 0.99 - fully saturated - and broke out at era 27 as the spread fell to 0.75. That is consistent with EITHER story. The user reports the Neutral phase on most runs, so it is worth four longs to stop guessing. Four per-era counters on the pass 3 OOS walk, reported as: | Neutral CHOSE 12.4% / TIED 38.1% (of which B=S 1204) | rail 61.2% m_oosNeutralStrict - Neutral strictly highest m_oosNeutralTie - no strict winner; the tie-break produced Neutral m_oosTieBuySell - the costly subset: Buy and Sell tied AT the top, i.e. a DIRECTIONAL reading thrown away by float equality m_oosRailBars - any raw output sitting on a sigmoid asymptote, the saturation that makes exact ties possible at all Read on the RAW logits, before ApplyClassificationSoftmax() overwrites TempData in place. Legitimate because softmax is strictly monotone: it cannot change the ordering and cannot break a tie either, so the raw reading and the decision always agree. Placed alongside the existing min/max/spread capture so all the output diagnostics describe the same values. Measurement only - no decision path reads these. NOT COMPILED - user compiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:06:54 -04:00
long m_oosNeutralStrict;
long m_oosNeutralTie;
long m_oosTieBuySell;
long m_oosRailBars; // any raw output pinned to a sigmoid rail (<=0+eps or >=1-eps)
//--- per-era counts of the network's own classification of each bar it fed forward (IS+OOS),
//--- reset at the start of every era; surfaced in the status label text so class imbalance
//--- (e.g. the network collapsing to all-Neutral) is visible while training runs
int m_countBuySignals;
int m_countSellSignals;
int m_countNeutralSignals;
//--- per-era counts of the *true* label of every bar fed forward (IS+OOS), reset alongside the
//--- predicted counts above.
int m_trueBuyCount;
int m_trueSellCount;
int m_trueNeutralCount;
//--- snapshot of the class totals above, taken at the end of the PREVIOUS era (see Train()'s
//--- era-reset block) and held fixed for the whole of the current era.
int m_prevEraTrueBuyCount;
int m_prevEraTrueSellCount;
int m_prevEraTrueNeutralCount;
//--- per-era OOS confusion counts, reset each era; used to compute per-class OOS recall (hits/total)
//--- for the status label text and, more importantly, as an additional convergence gate alongside the
//--- blended dOosForecast accuracy - a model that "wins" only by calling everything Neutral will
//--- have high dOosForecast but near-zero Buy/Sell recall, and should NOT be allowed to converge.
int m_oosBuyHits, m_oosBuyTotal;
int m_oosSellHits, m_oosSellTotal;
int m_oosNeutralHits, m_oosNeutralTotal;
//--- same per-era OOS confusion counts as above but keyed by PREDICTED class instead of true
//--- class, i.e. per-class precision (of the bars this era where the model called Sell, how many
//--- actually were Sell?) rather than recall (of the bars that actually were Sell, how many did
//--- it catch?).
int m_oosBuyPredicted, m_oosBuyPredictedHits;
int m_oosSellPredicted, m_oosSellPredictedHits;
int m_oosNeutralPredicted, m_oosNeutralPredictedHits;
//--- WOULD THE TRADE HAVE PAID - scored against the win caches rather than against label
//--- agreement: "Buy calls whose long actually reached target before stop", not "Buy calls that
//--- matched a 3-class label".
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
int m_oosBuyPredictedWins;
int m_oosSellPredictedWins;
//--- Zero-skill denominators, MEASURED rather than assumed from m/(m+k): how many scored bars a
//--- long (resp. a short) would have won on, regardless of what the model called or what the
//--- label says. max() of the two is what an always-call-one-direction model scores, which is
//--- the reference the deploy gate needs.
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
int m_oosWinLongTotal;
int m_oosWinShortTotal;
//--- Confidence calibration. EMA-blended across eras so one noisy era cannot swing it, and
//--- clamped to [0.3, 1.5] so a degenerate OOS window cannot drive it somewhere absurd.
double m_oosConfidenceSum;
double m_confidenceCalScale;
//--- minimum acceptable OOS recall (%) for the Buy and Sell classes individually before Train() is
//--- allowed to declare convergence; a class with zero OOS samples this era doesn't block (avoids a
//--- deadlock when a given era's OOS window happens to contain no examples of that class)
int m_minDirectionalRecallPct;
//--- THE OPERATIVE per-class floor, derived from the class's own effective sample (the input
//--- above is only the fallback when the sample is too thin for an SE).
feat(gates): derive the recall floor instead of configuring it, publish what the configuration can PROVE, and stop an arbitrary member driving live exits Three changes, all from the same principle: measure what is there before aiming at it, and never certify a number you do not trade. 1. THE RECALL FLOOR IS DERIVED, AND IT MOVES BELOW CHANCE. MinRecall=40 was a constant doing a statistical job. Its reference point is the 33.3% recall a zero-skill 3-class model gets on EVERY class, and against that the constant was accidentally calibrated for exactly one sample size: on USDJPY CONV (n_eff 195) 40% is chance + 2.0 SE; on SP500 PAI (n_eff 42) the same 40% is chance + 0.9 SE. One chart was being held to a bar twice as strict as the other, for no reason anyone chose. CollapseRecallFloorPct() computes it per class from that class's own effective sample - EffectiveSampleSize(), so the overlap deflation the rest of the gates use applies here too - as chance - EDGE_MIN_SIGMAS x SE. 26.5% at n_eff 195, 18.8% at n_eff 42. BELOW chance, deliberately, and this is the substantive change rather than the arithmetic. This gate's only job is refusing to call a COLLAPSED model converged. It is not a quality bar; the deploy gate is the quality bar and it is already rigorous (chance + 2 SE on the deflated sample, Sidak over candidate eras, then the cross-instrument pooled certificate). A convergence gate that ALSO demands provably-above-chance recall on all three classes double-counts that job, and it has failed that way twice here: MinRecall=60 blocked every SP500 H1 run in 2026-07, and the 40 that replaced it made Neutral structurally unreachable once first-touch resolution cut Neutral to a 0.65% residue. A floor nothing can reach does not make a funded account safer, it stops the run converging at all. Testing significantly BELOW chance instead catches what a fixed 40 was actually catching - a model that has stopped emitting a class - and cannot become unreachable by construction. It also fixes the direction the old constant scaled: it now widens on a thin OOS window, where low recall genuinely cannot be told from noise, and tightens on a rich one. Today's SP500 PAI (Buy 51 / Sell 18 / Neutral 30) is still correctly blocked on Sell. This also resolves a standing contradiction the code half-admitted at the isBetterEra comment: selection ranks on coverage-weighted PRECISION while convergence gated on RECALL, so a sparse high-precision abstainer - precisely the model that could clear the deploy bar - was blocked by the floor. The era line now PRINTS the derived floor. Anyone comparing these recalls against a remembered "40" is reading the wrong bar. 2. DETECTABILITY: WHAT THIS CONFIGURATION COULD PROVE, BEFORE IT TRAINS. The DEPLOY BAR line states the bar. It never said what reaching it would take, and that is the actionable direction. ReportDetectability() inverts the same identity - the gate passes when edge >= z x sqrt(p(1-p)/n_eff), so certifying an edge d needs n_eff >= z^2 p(1-p)/d^2 independent calls, hence L times as many raw ones - and prints a +2 / +5 / +10pp ladder as required independent calls, raw calls, and share of the OOS window, marking any rung that needs more than the window holds IMPOSSIBLE. Every term is a property of the CONFIGURATION - geometry via break-even, horizon via mean label lifespan, window via oosCutoff - so no amount of training moves any of them. It fires once, at the first healthy sweep, beside ReportFeatureHealth, for the same reason: that is the first moment the bar grid, the measured geometry and the lifespan are real numbers rather than defaults. It gates nothing. This is the EdgeFinder discipline applied to our own gate: establish what the market and the measurement design have to offer, then point the net at it - rather than spending a thousand eras chasing something this OOS window could never certify. 3. AN ARBITRARY MEMBER WAS DRIVING LIVE EXITS AND TRAILING (user-identified). Every ensemble member ran g_LiveAISignedConfidence = SignedAIConfidence(); unconditionally, every tick. Last writer wins. Its consumers are the AI early-exit route (CExpertSignalCustom::LiveSignedConfidence) and TrailingIntelligent - so on a four-model chart an LSTM entry could be closed, and its stop moved, on the Perceptron's opinion alone, decided by scheduling order. Not the vote, not a weighted blend. Now the mean across registered members, matching how the ensemble actually trades: the open decision is the weighted-average vote, and an abstaining member contributes 0 and dilutes exactly as it does there. Members still training read 0, so a half-trained ensemble reads WEAKER rather than louder - the safe direction for an exit trigger. Deployed and paused members are included, which is the opposite of the era barrier's exemption rule and correct for the opposite reason: that one asks who must be waited for, this asks who has an opinion. Latent today and staying that way for now by choice - Min_Vote_Close ships Disabled (101, unreachable on both scales it drives) and TrailingStrategy is off, so live exits are SL/TP only and the certified hold-to-barrier win rate is what actually gets traded. Fixed now precisely because the plan is to enable vote exits once the models are accurate, at which point a scheduling-order exit would be both harmful and very hard to see. STILL OPEN, and needs a decision before vote exits go on: the member gate and the ensemble vote gate both grade hold-to-barrier, so enabling vote exits makes the certified number stop describing the traded one. Warrior_EA.mq5 currently argues barrier models may keep vote exits because "their label IS the vote's own horizon" - that does not hold, since a vote flip at bar 5 of a 64-bar horizon is not the target-before-stop outcome the gate measured. Either grade the OOS call on the real exit rule (first of SL / TP / vote-flip / horizon) through the fill engine, or set HoldToBarrier for ensemble members so the policy cannot drift from the certificate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:15:08 -04:00
double CollapseRecallFloorPct(int classTrueCount);
//--- Last era-progress Print for THIS member (see the era loop's 5s rate limit).
uint m_lastProgressLogTick;
revert(labels): drop the one-sided exit target; measure the calibration drift instead Reverts a863796 on the operator's call - "unnecessary complexity". It was right about the mechanism and wrong about the priority: it re-cut the classes for a case the measured verdict never reaches (SP500 H4 reads "both sides" at the derived geometry), while the drift that IS happening affects every chart and every era. Recoverable from a863796 if a one-sided book ever becomes real. Two pieces of it survive, both independent of the exit idea: The drift verdict keeps reading m_winLongCache/m_winShortCache rather than the collapsed label pair. That line reports always-long vs always-short win rates, which is what the win caches hold - each side scored on its own barriers, published before the collapse. The label pair carries only the side touched first, so it undercounted long wins by the both-won-goes-to-short share. There are zero both-won bars at any geometry with target >= stop, so this changes no number today; it changes the wrong number to the right one. And the .cfg gains nothing and loses nothing: the two appended ints go away again, and they were the last fields, so a .cfg written by yesterday's build still reads correctly - the loader simply stops before them. WHAT THE REVERT MAKES ROOM FOR. The operator's actual requirement is that the model reproduce the label distribution the scan measured, and nothing in the pipeline ties it to that. The loss trains on a rebalanced sample and the abstain rate is owned by a margin threshold fitted on EDGE, so the call rate and the label prior can drift arbitrarily far apart - and did, invisibly: at era 1350 the models call Buy on 20-28% and Sell on 22-32% of bars against a scan-measured 2.1% and 4.8%. Roughly a 10x over-call, and not one line in the journal said so. The era line now carries it: CALIBRATION calls vs true rate Buy 28% vs 2% (14.0x) Sell 32% vs 5% (6.4x) Neutral 40% vs 93% (0.4x) Reported as a ratio because that is the readable number - 1.0x is calibrated. This is deliberately a measurement and not yet a correction: matching the label rate would put coverage near 7%, below the ensemble gate's own 12.4% coverage floor, so calibration and the gate are in direct conflict and which one yields is the operator's call, not mine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 10:14:26 -04:00
//--- THE EXIT POLICY ACTUALLY IN FORCE, pushed in from the same inputs the live path reads
//--- (Signal_ThresholdClose / HoldToBarrier). 0 = no vote-driven exit, which is what ships
//--- today.
feat(gate): grade OOS calls on the exit policy actually in force, and move vote combining out of the members and into the orchestrator Option (a) from the exit-policy question: the certified number must be the traded number. Plus the modularity correction the user called for on 778b6c0. 1. VOTE COMBINING BELONGS TO THE ORCHESTRATOR, NOT TO A MEMBER. 778b6c0 fixed the last-writer-wins bug on g_LiveAISignedConfidence by having a member average its siblings through g_warriorEnsemble. That trades a scheduling bug for a coupling bug, and it is the wrong shape for this EA: every signal runs in its own instance, minds its own state, and VOTES to the orchestrator, which is the only thing allowed to combine opinions. Replaced with a publish/aggregate pair in Variables\ConfidenceBridge.mqh whose split is enforced by shape rather than by convention: - PublishAIVote(slot, conf) - a member writes ONLY its own slot, reads nobody's; - AggregateAIVotes() - called by CExpertSignalCustom::LiveSignedConfidence. CExpertSignalAIBase::EnsembleLiveSignedConfidence is gone. The orchestrator also republishes the aggregate into g_LiveAISignedConfidence, because the intelligent trailing reads that global directly and must act on the same number the exit route does rather than on a leftover from whichever member ticked last. A solo AI signal owns slot 0, so the non-ensemble path is unchanged. 2. THE GATE NOW REPLAYS THE REAL EXIT RULE. SimulateTradeOutcome() walks the same price series with the same fill/barrier/spread convention as ComputeLabelForBar - deliberately by copy, so a disagreement between the two can only be a policy effect and never a discrepancy between two pieces of our own arithmetic - and terminates at the FIRST of stop / target / vote reversal / horizon. Barriers are tested before the vote on the same bar: intrabar we cannot know which came first, and the barrier is what the broker executes automatically, so checking the vote first would credit the exit policy with escapes a real stop would have taken out of its hands. It runs AFTER pass 3, not inside it. A vote-flip exit for a trade entered at bar r is decided by the model's output at bars r-1, r-2, ... - NEWER bars - and pass 3 walks oldest-to-newest, so at the moment r is graded its own exit does not exist yet. Only once m_oosDecisionSeries is complete over the whole OOS window can the replay run. In ensemble mode that series carries the member's adjusted decision and the live exit reads the ensemble aggregate, which is the coupling the user identified: an LSTM entry really can be closed by the ensemble turning against it. 3. THE STATISTIC HAS TO CHANGE WITH THE POLICY, AND THAT IS THE REAL FINDING. A barrier exit pays a fixed R. A vote exit pays whatever the close happens to be. So the moment vote exits are enabled the payoff is CONTINUOUS, and "win rate vs break-even" stops being a meaningful test - there is no fixed break-even for a variable payoff. SimulateTradeOutcome therefore returns R rather than a bool, and the replay reports expectancy in R with its SE taken from the R distribution (overlap- deflated on the same EffectiveSampleSize doctrine as every other SE here), not from a binomial. This is the same class of error as win-based scoring in 2026-08-09: measuring a variable-payoff process with a fixed-payoff statistic. Naming it now, while vote exits are still off, is much cheaper than discovering it after they go on. 4. WHY THIS IS SAFE TO SHIP TODAY. Min_Vote_Close ships Disabled (101, unreachable on both scales it drives) and reaches the AI signal through the new ExitPolicy() setter as 1.01, which the setter turns into 0.0 = no vote exit. Under that policy every replayed trade resolves at a barrier and the simulation is arithmetically the same trade the deploy gate already certifies - they cannot drift. The report says so explicitly, and prints ONCE per run in that state; when vote exits are on it prints every era, because then the divergence is the thing to watch. Nothing about today's numbers moves. The gate switchover is wired but dormant by construction: it becomes exit-aware the moment the input is enabled, which is exactly what "the certified number is the traded number" has to mean. KNOWN LIMIT, stated rather than hidden: only the AI early-exit route is replayed. The rule-based averaged-vote close (m_threshold_close) depends on every other filter's live vote, which pass 3 does not reproduce, so a position the classic filters would have closed is held to its barrier here. The replay therefore holds LONGER than live and overstates barrier-reached outcomes. Faithful only while the AI is the dominant vote - which is the configuration this is being built for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:25:57 -04:00
double m_exitVoteThreshold;
bool m_exitHoldToBarrier;
//--- The model's ADJUSTED signed decision per OOS bar, captured during pass 3 - the same value
//--- that votes live, 0 where it abstains.
feat(gate): grade OOS calls on the exit policy actually in force, and move vote combining out of the members and into the orchestrator Option (a) from the exit-policy question: the certified number must be the traded number. Plus the modularity correction the user called for on 778b6c0. 1. VOTE COMBINING BELONGS TO THE ORCHESTRATOR, NOT TO A MEMBER. 778b6c0 fixed the last-writer-wins bug on g_LiveAISignedConfidence by having a member average its siblings through g_warriorEnsemble. That trades a scheduling bug for a coupling bug, and it is the wrong shape for this EA: every signal runs in its own instance, minds its own state, and VOTES to the orchestrator, which is the only thing allowed to combine opinions. Replaced with a publish/aggregate pair in Variables\ConfidenceBridge.mqh whose split is enforced by shape rather than by convention: - PublishAIVote(slot, conf) - a member writes ONLY its own slot, reads nobody's; - AggregateAIVotes() - called by CExpertSignalCustom::LiveSignedConfidence. CExpertSignalAIBase::EnsembleLiveSignedConfidence is gone. The orchestrator also republishes the aggregate into g_LiveAISignedConfidence, because the intelligent trailing reads that global directly and must act on the same number the exit route does rather than on a leftover from whichever member ticked last. A solo AI signal owns slot 0, so the non-ensemble path is unchanged. 2. THE GATE NOW REPLAYS THE REAL EXIT RULE. SimulateTradeOutcome() walks the same price series with the same fill/barrier/spread convention as ComputeLabelForBar - deliberately by copy, so a disagreement between the two can only be a policy effect and never a discrepancy between two pieces of our own arithmetic - and terminates at the FIRST of stop / target / vote reversal / horizon. Barriers are tested before the vote on the same bar: intrabar we cannot know which came first, and the barrier is what the broker executes automatically, so checking the vote first would credit the exit policy with escapes a real stop would have taken out of its hands. It runs AFTER pass 3, not inside it. A vote-flip exit for a trade entered at bar r is decided by the model's output at bars r-1, r-2, ... - NEWER bars - and pass 3 walks oldest-to-newest, so at the moment r is graded its own exit does not exist yet. Only once m_oosDecisionSeries is complete over the whole OOS window can the replay run. In ensemble mode that series carries the member's adjusted decision and the live exit reads the ensemble aggregate, which is the coupling the user identified: an LSTM entry really can be closed by the ensemble turning against it. 3. THE STATISTIC HAS TO CHANGE WITH THE POLICY, AND THAT IS THE REAL FINDING. A barrier exit pays a fixed R. A vote exit pays whatever the close happens to be. So the moment vote exits are enabled the payoff is CONTINUOUS, and "win rate vs break-even" stops being a meaningful test - there is no fixed break-even for a variable payoff. SimulateTradeOutcome therefore returns R rather than a bool, and the replay reports expectancy in R with its SE taken from the R distribution (overlap- deflated on the same EffectiveSampleSize doctrine as every other SE here), not from a binomial. This is the same class of error as win-based scoring in 2026-08-09: measuring a variable-payoff process with a fixed-payoff statistic. Naming it now, while vote exits are still off, is much cheaper than discovering it after they go on. 4. WHY THIS IS SAFE TO SHIP TODAY. Min_Vote_Close ships Disabled (101, unreachable on both scales it drives) and reaches the AI signal through the new ExitPolicy() setter as 1.01, which the setter turns into 0.0 = no vote exit. Under that policy every replayed trade resolves at a barrier and the simulation is arithmetically the same trade the deploy gate already certifies - they cannot drift. The report says so explicitly, and prints ONCE per run in that state; when vote exits are on it prints every era, because then the divergence is the thing to watch. Nothing about today's numbers moves. The gate switchover is wired but dormant by construction: it becomes exit-aware the moment the input is enabled, which is exactly what "the certified number is the traded number" has to mean. KNOWN LIMIT, stated rather than hidden: only the AI early-exit route is replayed. The rule-based averaged-vote close (m_threshold_close) depends on every other filter's live vote, which pass 3 does not reproduce, so a position the classic filters would have closed is held to its barrier here. The replay therefore holds LONGER than live and overstates barrier-reached outcomes. Faithful only while the AI is the dominant vote - which is the configuration this is being built for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:25:57 -04:00
double m_oosDecisionSeries[];
//--- The trade the EA would ACTUALLY have taken from `entryIdx`, under the policy above: first
//--- of stop / target / vote reversal / horizon.
feat(gate): grade OOS calls on the exit policy actually in force, and move vote combining out of the members and into the orchestrator Option (a) from the exit-policy question: the certified number must be the traded number. Plus the modularity correction the user called for on 778b6c0. 1. VOTE COMBINING BELONGS TO THE ORCHESTRATOR, NOT TO A MEMBER. 778b6c0 fixed the last-writer-wins bug on g_LiveAISignedConfidence by having a member average its siblings through g_warriorEnsemble. That trades a scheduling bug for a coupling bug, and it is the wrong shape for this EA: every signal runs in its own instance, minds its own state, and VOTES to the orchestrator, which is the only thing allowed to combine opinions. Replaced with a publish/aggregate pair in Variables\ConfidenceBridge.mqh whose split is enforced by shape rather than by convention: - PublishAIVote(slot, conf) - a member writes ONLY its own slot, reads nobody's; - AggregateAIVotes() - called by CExpertSignalCustom::LiveSignedConfidence. CExpertSignalAIBase::EnsembleLiveSignedConfidence is gone. The orchestrator also republishes the aggregate into g_LiveAISignedConfidence, because the intelligent trailing reads that global directly and must act on the same number the exit route does rather than on a leftover from whichever member ticked last. A solo AI signal owns slot 0, so the non-ensemble path is unchanged. 2. THE GATE NOW REPLAYS THE REAL EXIT RULE. SimulateTradeOutcome() walks the same price series with the same fill/barrier/spread convention as ComputeLabelForBar - deliberately by copy, so a disagreement between the two can only be a policy effect and never a discrepancy between two pieces of our own arithmetic - and terminates at the FIRST of stop / target / vote reversal / horizon. Barriers are tested before the vote on the same bar: intrabar we cannot know which came first, and the barrier is what the broker executes automatically, so checking the vote first would credit the exit policy with escapes a real stop would have taken out of its hands. It runs AFTER pass 3, not inside it. A vote-flip exit for a trade entered at bar r is decided by the model's output at bars r-1, r-2, ... - NEWER bars - and pass 3 walks oldest-to-newest, so at the moment r is graded its own exit does not exist yet. Only once m_oosDecisionSeries is complete over the whole OOS window can the replay run. In ensemble mode that series carries the member's adjusted decision and the live exit reads the ensemble aggregate, which is the coupling the user identified: an LSTM entry really can be closed by the ensemble turning against it. 3. THE STATISTIC HAS TO CHANGE WITH THE POLICY, AND THAT IS THE REAL FINDING. A barrier exit pays a fixed R. A vote exit pays whatever the close happens to be. So the moment vote exits are enabled the payoff is CONTINUOUS, and "win rate vs break-even" stops being a meaningful test - there is no fixed break-even for a variable payoff. SimulateTradeOutcome therefore returns R rather than a bool, and the replay reports expectancy in R with its SE taken from the R distribution (overlap- deflated on the same EffectiveSampleSize doctrine as every other SE here), not from a binomial. This is the same class of error as win-based scoring in 2026-08-09: measuring a variable-payoff process with a fixed-payoff statistic. Naming it now, while vote exits are still off, is much cheaper than discovering it after they go on. 4. WHY THIS IS SAFE TO SHIP TODAY. Min_Vote_Close ships Disabled (101, unreachable on both scales it drives) and reaches the AI signal through the new ExitPolicy() setter as 1.01, which the setter turns into 0.0 = no vote exit. Under that policy every replayed trade resolves at a barrier and the simulation is arithmetically the same trade the deploy gate already certifies - they cannot drift. The report says so explicitly, and prints ONCE per run in that state; when vote exits are on it prints every era, because then the divergence is the thing to watch. Nothing about today's numbers moves. The gate switchover is wired but dormant by construction: it becomes exit-aware the moment the input is enabled, which is exactly what "the certified number is the traded number" has to mean. KNOWN LIMIT, stated rather than hidden: only the AI early-exit route is replayed. The rule-based averaged-vote close (m_threshold_close) depends on every other filter's live vote, which pass 3 does not reproduce, so a position the classic filters would have closed is held to its barrier here. The replay therefore holds LONGER than live and overstates barrier-reached outcomes. Faithful only while the AI is the dominant vote - which is the configuration this is being built for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:25:57 -04:00
bool SimulateTradeOutcome(int entryIdx, bool isLong, double &rMultiple,
feat(breakeven): the break-even every layer scores against prices a trade that always resolves CostAdjustedBreakEvenPct is risk/(risk+reward) and has no horizon term. It is the win rate a trade needs when it is CERTAIN to end at one barrier or the other. SimulateTradeOutcome has an explicit branch for the case where it does not - runs out of horizon, closes at the last bar seen for whatever P&L that is - so on this label geometry the figure describes a different trade than the one being replayed. The gap is measurable and large. Across 21 exit replays today on SP500 H4 the geometric figure read 34.5% while the EA's own R simulation crossed zero between 27.4% (lowest positive) and 28.9% (highest non-positive). Independent corroboration: the zero-skill reference, computed empirically over every scored bar as max(winLong,winShort)/bars, reads 25.4% - add cost and it lands on the same ~28%. The geometric number is the outlier, and every edge printed against it was ~6.5pp too pessimistic: LSTM's 30.6%-win era reported -4.0pp while its replay returned +0.075 R on the same trades. With a timeout share t paying a mean m R apiece, expectancy is w(1+RR) + t(1+m) - 1, so w* = (1 - t(1+m)) / (1 + RR) = CostAdjustedBreakEvenPct x (1 - t(1+m)) which needs no new geometry - the existing figure already carries 1/(1+RR). This commit MEASURES ONLY. The replay now separates timeout exits from barrier exits and latches t and m for the next era to read (the accumulators are zeroed at era start and filled at era end, so a mid-era reader sees zero trades and would fall back forever). Both break-evens print side by side on the replay line with t and m beside them, and the threshold line's REPORTED edge - which selects nothing - switches to the horizon-aware figure so the operator stops reading a wrong sign. DELIBERATELY NOT CHANGED: LiveMetaGate's veto and the rung selector's BarrierMinReachPct still read the geometric value. Both are decisions - the second re-derives geometry and therefore relabels - and t and m have so far only been inferred from a zero-crossing, never seen on a log. One era of this instrumentation settles that. The file already contained the argument, one branch away, in the vote-exit comment: a vote exit produces a CONTINUOUS payoff, not a win or a loss, and that is why an exit-aware gate cannot go on scoring win-rate against a fixed break-even. A horizon timeout is the same thing, and unlike vote exits it is on by default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 19:26:25 -04:00
int &lifespanBars, bool &endedOnVote,
bool &endedOnTimeout);
feat(gate): grade OOS calls on the exit policy actually in force, and move vote combining out of the members and into the orchestrator Option (a) from the exit-policy question: the certified number must be the traded number. Plus the modularity correction the user called for on 778b6c0. 1. VOTE COMBINING BELONGS TO THE ORCHESTRATOR, NOT TO A MEMBER. 778b6c0 fixed the last-writer-wins bug on g_LiveAISignedConfidence by having a member average its siblings through g_warriorEnsemble. That trades a scheduling bug for a coupling bug, and it is the wrong shape for this EA: every signal runs in its own instance, minds its own state, and VOTES to the orchestrator, which is the only thing allowed to combine opinions. Replaced with a publish/aggregate pair in Variables\ConfidenceBridge.mqh whose split is enforced by shape rather than by convention: - PublishAIVote(slot, conf) - a member writes ONLY its own slot, reads nobody's; - AggregateAIVotes() - called by CExpertSignalCustom::LiveSignedConfidence. CExpertSignalAIBase::EnsembleLiveSignedConfidence is gone. The orchestrator also republishes the aggregate into g_LiveAISignedConfidence, because the intelligent trailing reads that global directly and must act on the same number the exit route does rather than on a leftover from whichever member ticked last. A solo AI signal owns slot 0, so the non-ensemble path is unchanged. 2. THE GATE NOW REPLAYS THE REAL EXIT RULE. SimulateTradeOutcome() walks the same price series with the same fill/barrier/spread convention as ComputeLabelForBar - deliberately by copy, so a disagreement between the two can only be a policy effect and never a discrepancy between two pieces of our own arithmetic - and terminates at the FIRST of stop / target / vote reversal / horizon. Barriers are tested before the vote on the same bar: intrabar we cannot know which came first, and the barrier is what the broker executes automatically, so checking the vote first would credit the exit policy with escapes a real stop would have taken out of its hands. It runs AFTER pass 3, not inside it. A vote-flip exit for a trade entered at bar r is decided by the model's output at bars r-1, r-2, ... - NEWER bars - and pass 3 walks oldest-to-newest, so at the moment r is graded its own exit does not exist yet. Only once m_oosDecisionSeries is complete over the whole OOS window can the replay run. In ensemble mode that series carries the member's adjusted decision and the live exit reads the ensemble aggregate, which is the coupling the user identified: an LSTM entry really can be closed by the ensemble turning against it. 3. THE STATISTIC HAS TO CHANGE WITH THE POLICY, AND THAT IS THE REAL FINDING. A barrier exit pays a fixed R. A vote exit pays whatever the close happens to be. So the moment vote exits are enabled the payoff is CONTINUOUS, and "win rate vs break-even" stops being a meaningful test - there is no fixed break-even for a variable payoff. SimulateTradeOutcome therefore returns R rather than a bool, and the replay reports expectancy in R with its SE taken from the R distribution (overlap- deflated on the same EffectiveSampleSize doctrine as every other SE here), not from a binomial. This is the same class of error as win-based scoring in 2026-08-09: measuring a variable-payoff process with a fixed-payoff statistic. Naming it now, while vote exits are still off, is much cheaper than discovering it after they go on. 4. WHY THIS IS SAFE TO SHIP TODAY. Min_Vote_Close ships Disabled (101, unreachable on both scales it drives) and reaches the AI signal through the new ExitPolicy() setter as 1.01, which the setter turns into 0.0 = no vote exit. Under that policy every replayed trade resolves at a barrier and the simulation is arithmetically the same trade the deploy gate already certifies - they cannot drift. The report says so explicitly, and prints ONCE per run in that state; when vote exits are on it prints every era, because then the divergence is the thing to watch. Nothing about today's numbers moves. The gate switchover is wired but dormant by construction: it becomes exit-aware the moment the input is enabled, which is exactly what "the certified number is the traded number" has to mean. KNOWN LIMIT, stated rather than hidden: only the AI early-exit route is replayed. The rule-based averaged-vote close (m_threshold_close) depends on every other filter's live vote, which pass 3 does not reproduce, so a position the classic filters would have closed is held to its barrier here. The replay therefore holds LONGER than live and overstates barrier-reached outcomes. Faithful only while the AI is the dominant vote - which is the configuration this is being built for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:25:57 -04:00
//--- Per-era accumulators for the simulated-exit report (see ReportExitPolicyDivergence).
double m_simRSum;
double m_simRSumSq;
feat(breakeven): the break-even every layer scores against prices a trade that always resolves CostAdjustedBreakEvenPct is risk/(risk+reward) and has no horizon term. It is the win rate a trade needs when it is CERTAIN to end at one barrier or the other. SimulateTradeOutcome has an explicit branch for the case where it does not - runs out of horizon, closes at the last bar seen for whatever P&L that is - so on this label geometry the figure describes a different trade than the one being replayed. The gap is measurable and large. Across 21 exit replays today on SP500 H4 the geometric figure read 34.5% while the EA's own R simulation crossed zero between 27.4% (lowest positive) and 28.9% (highest non-positive). Independent corroboration: the zero-skill reference, computed empirically over every scored bar as max(winLong,winShort)/bars, reads 25.4% - add cost and it lands on the same ~28%. The geometric number is the outlier, and every edge printed against it was ~6.5pp too pessimistic: LSTM's 30.6%-win era reported -4.0pp while its replay returned +0.075 R on the same trades. With a timeout share t paying a mean m R apiece, expectancy is w(1+RR) + t(1+m) - 1, so w* = (1 - t(1+m)) / (1 + RR) = CostAdjustedBreakEvenPct x (1 - t(1+m)) which needs no new geometry - the existing figure already carries 1/(1+RR). This commit MEASURES ONLY. The replay now separates timeout exits from barrier exits and latches t and m for the next era to read (the accumulators are zeroed at era start and filled at era end, so a mid-era reader sees zero trades and would fall back forever). Both break-evens print side by side on the replay line with t and m beside them, and the threshold line's REPORTED edge - which selects nothing - switches to the horizon-aware figure so the operator stops reading a wrong sign. DELIBERATELY NOT CHANGED: LiveMetaGate's veto and the rung selector's BarrierMinReachPct still read the geometric value. Both are decisions - the second re-derives geometry and therefore relabels - and t and m have so far only been inferred from a zero-crossing, never seen on a log. One era of this instrumentation settles that. The file already contained the argument, one branch away, in the vote-exit comment: a vote exit produces a CONTINUOUS payoff, not a win or a loss, and that is why an exit-aware gate cannot go on scoring win-rate against a fixed break-even. A horizon timeout is the same thing, and unlike vote exits it is on by default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 19:26:25 -04:00
//--- Trades that reached NEITHER barrier inside the horizon, and what they actually paid. These
//--- are what CostAdjustedBreakEvenPct does not know about - see EmpiricalBreakEvenPct.
int m_simTimeouts;
double m_simTimeoutRSum;
//--- The LAST COMPLETED era's timeout measurement. Era N therefore reports against era N-1's
//--- measurement, which is the honest pairing anyway: a partial era is not one.
feat(breakeven): the break-even every layer scores against prices a trade that always resolves CostAdjustedBreakEvenPct is risk/(risk+reward) and has no horizon term. It is the win rate a trade needs when it is CERTAIN to end at one barrier or the other. SimulateTradeOutcome has an explicit branch for the case where it does not - runs out of horizon, closes at the last bar seen for whatever P&L that is - so on this label geometry the figure describes a different trade than the one being replayed. The gap is measurable and large. Across 21 exit replays today on SP500 H4 the geometric figure read 34.5% while the EA's own R simulation crossed zero between 27.4% (lowest positive) and 28.9% (highest non-positive). Independent corroboration: the zero-skill reference, computed empirically over every scored bar as max(winLong,winShort)/bars, reads 25.4% - add cost and it lands on the same ~28%. The geometric number is the outlier, and every edge printed against it was ~6.5pp too pessimistic: LSTM's 30.6%-win era reported -4.0pp while its replay returned +0.075 R on the same trades. With a timeout share t paying a mean m R apiece, expectancy is w(1+RR) + t(1+m) - 1, so w* = (1 - t(1+m)) / (1 + RR) = CostAdjustedBreakEvenPct x (1 - t(1+m)) which needs no new geometry - the existing figure already carries 1/(1+RR). This commit MEASURES ONLY. The replay now separates timeout exits from barrier exits and latches t and m for the next era to read (the accumulators are zeroed at era start and filled at era end, so a mid-era reader sees zero trades and would fall back forever). Both break-evens print side by side on the replay line with t and m beside them, and the threshold line's REPORTED edge - which selects nothing - switches to the horizon-aware figure so the operator stops reading a wrong sign. DELIBERATELY NOT CHANGED: LiveMetaGate's veto and the rung selector's BarrierMinReachPct still read the geometric value. Both are decisions - the second re-derives geometry and therefore relabels - and t and m have so far only been inferred from a zero-crossing, never seen on a log. One era of this instrumentation settles that. The file already contained the argument, one branch away, in the vote-exit comment: a vote exit produces a CONTINUOUS payoff, not a win or a loss, and that is why an exit-aware gate cannot go on scoring win-rate against a fixed break-even. A horizon timeout is the same thing, and unlike vote exits it is on by default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 19:26:25 -04:00
double m_lastTimeoutShare; // t in 0..1; -1 = never measured
double m_lastTimeoutMeanR; // mean R of the trades that timed out
feat(gate): grade OOS calls on the exit policy actually in force, and move vote combining out of the members and into the orchestrator Option (a) from the exit-policy question: the certified number must be the traded number. Plus the modularity correction the user called for on 778b6c0. 1. VOTE COMBINING BELONGS TO THE ORCHESTRATOR, NOT TO A MEMBER. 778b6c0 fixed the last-writer-wins bug on g_LiveAISignedConfidence by having a member average its siblings through g_warriorEnsemble. That trades a scheduling bug for a coupling bug, and it is the wrong shape for this EA: every signal runs in its own instance, minds its own state, and VOTES to the orchestrator, which is the only thing allowed to combine opinions. Replaced with a publish/aggregate pair in Variables\ConfidenceBridge.mqh whose split is enforced by shape rather than by convention: - PublishAIVote(slot, conf) - a member writes ONLY its own slot, reads nobody's; - AggregateAIVotes() - called by CExpertSignalCustom::LiveSignedConfidence. CExpertSignalAIBase::EnsembleLiveSignedConfidence is gone. The orchestrator also republishes the aggregate into g_LiveAISignedConfidence, because the intelligent trailing reads that global directly and must act on the same number the exit route does rather than on a leftover from whichever member ticked last. A solo AI signal owns slot 0, so the non-ensemble path is unchanged. 2. THE GATE NOW REPLAYS THE REAL EXIT RULE. SimulateTradeOutcome() walks the same price series with the same fill/barrier/spread convention as ComputeLabelForBar - deliberately by copy, so a disagreement between the two can only be a policy effect and never a discrepancy between two pieces of our own arithmetic - and terminates at the FIRST of stop / target / vote reversal / horizon. Barriers are tested before the vote on the same bar: intrabar we cannot know which came first, and the barrier is what the broker executes automatically, so checking the vote first would credit the exit policy with escapes a real stop would have taken out of its hands. It runs AFTER pass 3, not inside it. A vote-flip exit for a trade entered at bar r is decided by the model's output at bars r-1, r-2, ... - NEWER bars - and pass 3 walks oldest-to-newest, so at the moment r is graded its own exit does not exist yet. Only once m_oosDecisionSeries is complete over the whole OOS window can the replay run. In ensemble mode that series carries the member's adjusted decision and the live exit reads the ensemble aggregate, which is the coupling the user identified: an LSTM entry really can be closed by the ensemble turning against it. 3. THE STATISTIC HAS TO CHANGE WITH THE POLICY, AND THAT IS THE REAL FINDING. A barrier exit pays a fixed R. A vote exit pays whatever the close happens to be. So the moment vote exits are enabled the payoff is CONTINUOUS, and "win rate vs break-even" stops being a meaningful test - there is no fixed break-even for a variable payoff. SimulateTradeOutcome therefore returns R rather than a bool, and the replay reports expectancy in R with its SE taken from the R distribution (overlap- deflated on the same EffectiveSampleSize doctrine as every other SE here), not from a binomial. This is the same class of error as win-based scoring in 2026-08-09: measuring a variable-payoff process with a fixed-payoff statistic. Naming it now, while vote exits are still off, is much cheaper than discovering it after they go on. 4. WHY THIS IS SAFE TO SHIP TODAY. Min_Vote_Close ships Disabled (101, unreachable on both scales it drives) and reaches the AI signal through the new ExitPolicy() setter as 1.01, which the setter turns into 0.0 = no vote exit. Under that policy every replayed trade resolves at a barrier and the simulation is arithmetically the same trade the deploy gate already certifies - they cannot drift. The report says so explicitly, and prints ONCE per run in that state; when vote exits are on it prints every era, because then the divergence is the thing to watch. Nothing about today's numbers moves. The gate switchover is wired but dormant by construction: it becomes exit-aware the moment the input is enabled, which is exactly what "the certified number is the traded number" has to mean. KNOWN LIMIT, stated rather than hidden: only the AI early-exit route is replayed. The rule-based averaged-vote close (m_threshold_close) depends on every other filter's live vote, which pass 3 does not reproduce, so a position the classic filters would have closed is held to its barrier here. The replay therefore holds LONGER than live and overstates barrier-reached outcomes. Faithful only while the AI is the dominant vote - which is the configuration this is being built for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:25:57 -04:00
int m_simTrades;
int m_simVoteExits;
int m_simBarrierWins;
fix(replay): the exit replay held trades through the Friday flat that the label and the live EA both close The EXIT-POLICY REPLAY line reported an expectancy from SimulateTradeOutcome beside a win rate read out of the label cache, and called them "the SAME calls". Same calls, two different walks - and the walks did not agree. TripleBarrierLabel stops at NextScheduledCloseAll (3e467f9); SimulateTradeOutcome never called it, so the replay kept holding positions the live EA is flattened out of and collected targets the label had already scored as cut. On SP500 H4 the simulation's implied win rate ran 2.2-3.4pp above the label's on identical calls, and the timeout share read 0.8-1.3% because nothing was truncating the horizon it walked. That gap, plus 1.4pp of spread charged twice in CostAdjustedBreakEvenPct, is the whole of the ~5pp the replay looked "off" by. It was not horizon timeouts, which is what 4070c5c argued and this log disproved: solving E[R] = 3.008w - 1 + t(1+m) on each row puts the simulation's zero-crossing at an implied 33.3% against a frictionless 33.24% - it was internally consistent all along. - SimulateTradeOutcome takes the close-all cutoff, same expression and same placement as the label's, falling through to the existing close-at-last-bar branch. Expect the timeout share to rise and expectancy to fall: the replay was optimistic. - m_simTpHits counts this walk's own target-before-stop, printed next to the label's with the delta, so a future divergence is visible rather than inferable. - The line prints all three break-evens and names the R convention. The frictionless figure is the one this expectancy crosses zero at, because both walks place the barriers off the spread-shifted fill. - CostAdjustedBreakEvenPct is left alone: it still feeds the rung selector's BarrierMinReachPct, and moving that relabels. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 21:08:26 -04:00
//--- The SIMULATION's own target-before-stop count, beside the LABEL's above. Two walks, and the
//--- report used to quote only the label's while quoting the simulation's expectancy - so a
//--- disagreement between them read as a property of the trade rather than of our arithmetic.
int m_simTpHits;
feat(geometry): measure per-candidate barriers against the one global pair Stage 2a of the candidate-conditional geometry the record has named as next and never built. MEASUREMENT ONLY - no order uses it yet. WHY THIS AND NOT META-LABELING. Meta-labeling asks take-it-or-skip-it at fixed geometry, and its verdict stands: real skill, 0 operating points clearing break-even, and the 4070c5c retraction only moved that bar 1.4pp. The excursion head, by contrast, just cleared at 3.5-4.6 sigma on LSTM across four eras and beat the trailing-quantile incumbent. What is learnable here is MAGNITUDE, so the lever is the geometry, not the veto. A per-candidate rung means a per-candidate break-even, which a binary gate cannot express. HOW IT IS SCORED. At each OOS call the same bar is resolved under the incumbent pair AND under the pair this bar's excursion head would choose, and the paired difference is accumulated in R with a 2-sigma test. Both legs come from the SAME first-passage ladder - four array reads, no re-walk, exact even on the ~28% of bars where both barriers were touched. Mixing the ladder with the price walk here would measure the discrepancy between two of our own evaluators rather than the effect of the geometry, which is precisely what f8ac10c had to unpick one layer over. The candidate pair applies the GLOBAL derivation's own rule per bar: stop at BARRIER_SL_QUANTILE of adverse travel, target at the median of favourable. Neither creates expectancy; what moves is the break-even, which is why the report quotes R and never a win rate. FREE VALIDATION. The sigma gate predicts LSTM helps and CONV hurts. If the R difference reproduces that ordering across members, the head's usefulness is confirmed by a second, independent measurement. If it does not, something is wrong and this must not be wired to orders. Two things caught while writing it, both silent if missed: - The ladder stores TRAVEL FROM ENTRY, and the scan's mapping is risk = ladder + spread but reward = ladder - spread, so the two legs convert with OPPOSITE signs. The stop leg had the sign backwards. - A GEOMETRY_BUDGET_MS wall clock, because this adds a feature-window build and a head forward per OOS call to a walk that already runs unchunked at era end on a single-threaded EA. That is the shape that got the process force- terminated on 2026-08-21. It stops scoring, never the replay, and the report prints how many calls it covered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 08:29:59 -04:00
//--- CANDIDATE-CONDITIONAL GEOMETRY, measured only (see ReportCandidateGeometry). Paired per OOS
//--- call: the same bar scored under the global pair and under the pair this bar's excursion head
//--- would have chosen, both resolved from the SAME first-passage ladder so the difference cannot
//--- be an artifact of two evaluators disagreeing - the f8ac10c mistake, one layer over.
double m_geoDiffSum;
double m_geoDiffSumSq;
double m_geoIncSum;
double m_geoCandSum;
int m_geoTrades;
int m_geoIncOpen; // unresolved inside the horizon, incumbent pair
int m_geoCandOpen; // ... and candidate pair; a resolution gap skews the means
double m_geoCandSl; // running mean of the chosen multiples, for the report
double m_geoCandTp;
//--- Wall-clock guard. This adds a feature-window build and a head forward PER OOS call to a walk
//--- that already runs unchunked at era end, on a single-threaded EA - the shape that got the
//--- process force-terminated on 2026-08-21. It stops scoring rather than stopping the walk, so the
//--- replay above is unaffected and the report says how many calls it covered.
uint m_geoStartTick;
//--- Nearest ladder rung to a travel distance, in LOG space: the ladder is geometric, so a linear
//--- nearest biases every choice toward the coarse upper end.
int LadderRungFor(const double travelAtr);
//--- Exact outcome of one bar at one (stop, target) rung pair, in R. Four array reads, no re-walk.
bool LadderOutcomeR(const int barIdx, const bool isLong, const int slRung,
fix(geometry): a free zero made "never resolve" the winning geometry The CANDIDATE GEOMETRY line shipped in 05f1a53 said per-candidate geometry beats the global pair on every SP500 member at 2-3 sigma. It does not. It said so because a bar that reached neither barrier scored 0 R, and the incumbent's mean is NEGATIVE (-0.07 to -0.21 R). Against a losing baseline a free zero is a win, so the widest candidate always came out ahead - and the reported gain ordered itself by timeout share, not by skill: PAI 95.1% timed out -> +0.189 R (head measured -2.42 sigma, HARMFUL) HYB 73.8% -> +0.182 R (head at chance, +0.68 sigma) CONV 61.8% -> +0.163 R (head measured -2.47 sigma, HARMFUL) LSTM 27.1% -> +0.158 R (head +1.67 sigma) Monotone in the timeout share and inverted against the sigma gate. The acceptance test written when this was built - "the sigma gate predicts LSTM helps and CONV hurts; if the R difference does not reproduce that ordering, something is wrong" - is what caught it. A trade that reaches neither barrier is not worth zero. It is closed at the horizon, which is what the scheduled close-all does live and what SimulateTradeOutcome's timeout path already charges. So mark it there: TripleBarrierLabel now publishes the signed close-to-close travel at the last bar it actually visited (m_termTravelCache, same validity flag as the excursion and ladder caches), and LadderOutcomeR prices a timeout off it instead of returning false. A bar that cannot be evaluated under BOTH pairs is now dropped whole - scoring one leg and defaulting the other is the same bug in a smaller costume. Second defect, same function: CandidateGeometryFor applied neither of the floors the global derivation applies, so on USDJPY it chose stop 2.00 / target 1.00 - a 67% break-even, forbidden by the 1:2 policy floor. c3daded in miniature: a selector optimising its own criterion with no reference to the decision criterion. Both floors now apply, and the ratio is re-checked AFTER the per-leg rung snap, which can lose it. Also: the module weight was an unshrunk pooled win rate. USDJPY ConvLSTM fired 19 times (2.0 effective), won 36.8%, and took module weight 0.37 - 41% of the ensemble's capable weight and the loudest voice on the chart, off two effective observations. It also lifted the computed vote ceiling to 26.3 against a 25 threshold, which is why THRESHOLD UNREACHABLE never printed on a chart whose peak vote is 14 and whose practical ceiling without that member is 18.8. The pooled rate is now shrunk toward the coin-flip rate on the era's own OOS bars over 30 prior-equivalent calls, and the tiers shrink toward the shrunk value rather than the raw one. A member with ~300 effective calls moves by ~0.4pp; the 19-fire member goes 0.37 -> ~0.15. MEASUREMENT ONLY still - no order reads any of this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 11:30:54 -04:00
const int tpRung, double &rMultiple, bool &timedOut);
feat(geometry): measure per-candidate barriers against the one global pair Stage 2a of the candidate-conditional geometry the record has named as next and never built. MEASUREMENT ONLY - no order uses it yet. WHY THIS AND NOT META-LABELING. Meta-labeling asks take-it-or-skip-it at fixed geometry, and its verdict stands: real skill, 0 operating points clearing break-even, and the 4070c5c retraction only moved that bar 1.4pp. The excursion head, by contrast, just cleared at 3.5-4.6 sigma on LSTM across four eras and beat the trailing-quantile incumbent. What is learnable here is MAGNITUDE, so the lever is the geometry, not the veto. A per-candidate rung means a per-candidate break-even, which a binary gate cannot express. HOW IT IS SCORED. At each OOS call the same bar is resolved under the incumbent pair AND under the pair this bar's excursion head would choose, and the paired difference is accumulated in R with a 2-sigma test. Both legs come from the SAME first-passage ladder - four array reads, no re-walk, exact even on the ~28% of bars where both barriers were touched. Mixing the ladder with the price walk here would measure the discrepancy between two of our own evaluators rather than the effect of the geometry, which is precisely what f8ac10c had to unpick one layer over. The candidate pair applies the GLOBAL derivation's own rule per bar: stop at BARRIER_SL_QUANTILE of adverse travel, target at the median of favourable. Neither creates expectancy; what moves is the break-even, which is why the report quotes R and never a win rate. FREE VALIDATION. The sigma gate predicts LSTM helps and CONV hurts. If the R difference reproduces that ordering across members, the head's usefulness is confirmed by a second, independent measurement. If it does not, something is wrong and this must not be wired to orders. Two things caught while writing it, both silent if missed: - The ladder stores TRAVEL FROM ENTRY, and the scan's mapping is risk = ladder + spread but reward = ladder - spread, so the two legs convert with OPPOSITE signs. The stop leg had the sign backwards. - A GEOMETRY_BUDGET_MS wall clock, because this adds a feature-window build and a head forward per OOS call to a walk that already runs unchunked at era end on a single-threaded EA. That is the shape that got the process force- terminated on 2026-08-21. It stops scoring, never the replay, and the report prints how many calls it covered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 08:29:59 -04:00
//--- The pair this bar's excursion head would choose. False when the head cannot answer.
bool CandidateGeometryFor(const int barIdx, const bool isLong,
int &slRung, int &tpRung);
void ScoreCandidateGeometry(const int barIdx, const bool isLong);
void ReportCandidateGeometry(void);
feat(gate): grade OOS calls on the exit policy actually in force, and move vote combining out of the members and into the orchestrator Option (a) from the exit-policy question: the certified number must be the traded number. Plus the modularity correction the user called for on 778b6c0. 1. VOTE COMBINING BELONGS TO THE ORCHESTRATOR, NOT TO A MEMBER. 778b6c0 fixed the last-writer-wins bug on g_LiveAISignedConfidence by having a member average its siblings through g_warriorEnsemble. That trades a scheduling bug for a coupling bug, and it is the wrong shape for this EA: every signal runs in its own instance, minds its own state, and VOTES to the orchestrator, which is the only thing allowed to combine opinions. Replaced with a publish/aggregate pair in Variables\ConfidenceBridge.mqh whose split is enforced by shape rather than by convention: - PublishAIVote(slot, conf) - a member writes ONLY its own slot, reads nobody's; - AggregateAIVotes() - called by CExpertSignalCustom::LiveSignedConfidence. CExpertSignalAIBase::EnsembleLiveSignedConfidence is gone. The orchestrator also republishes the aggregate into g_LiveAISignedConfidence, because the intelligent trailing reads that global directly and must act on the same number the exit route does rather than on a leftover from whichever member ticked last. A solo AI signal owns slot 0, so the non-ensemble path is unchanged. 2. THE GATE NOW REPLAYS THE REAL EXIT RULE. SimulateTradeOutcome() walks the same price series with the same fill/barrier/spread convention as ComputeLabelForBar - deliberately by copy, so a disagreement between the two can only be a policy effect and never a discrepancy between two pieces of our own arithmetic - and terminates at the FIRST of stop / target / vote reversal / horizon. Barriers are tested before the vote on the same bar: intrabar we cannot know which came first, and the barrier is what the broker executes automatically, so checking the vote first would credit the exit policy with escapes a real stop would have taken out of its hands. It runs AFTER pass 3, not inside it. A vote-flip exit for a trade entered at bar r is decided by the model's output at bars r-1, r-2, ... - NEWER bars - and pass 3 walks oldest-to-newest, so at the moment r is graded its own exit does not exist yet. Only once m_oosDecisionSeries is complete over the whole OOS window can the replay run. In ensemble mode that series carries the member's adjusted decision and the live exit reads the ensemble aggregate, which is the coupling the user identified: an LSTM entry really can be closed by the ensemble turning against it. 3. THE STATISTIC HAS TO CHANGE WITH THE POLICY, AND THAT IS THE REAL FINDING. A barrier exit pays a fixed R. A vote exit pays whatever the close happens to be. So the moment vote exits are enabled the payoff is CONTINUOUS, and "win rate vs break-even" stops being a meaningful test - there is no fixed break-even for a variable payoff. SimulateTradeOutcome therefore returns R rather than a bool, and the replay reports expectancy in R with its SE taken from the R distribution (overlap- deflated on the same EffectiveSampleSize doctrine as every other SE here), not from a binomial. This is the same class of error as win-based scoring in 2026-08-09: measuring a variable-payoff process with a fixed-payoff statistic. Naming it now, while vote exits are still off, is much cheaper than discovering it after they go on. 4. WHY THIS IS SAFE TO SHIP TODAY. Min_Vote_Close ships Disabled (101, unreachable on both scales it drives) and reaches the AI signal through the new ExitPolicy() setter as 1.01, which the setter turns into 0.0 = no vote exit. Under that policy every replayed trade resolves at a barrier and the simulation is arithmetically the same trade the deploy gate already certifies - they cannot drift. The report says so explicitly, and prints ONCE per run in that state; when vote exits are on it prints every era, because then the divergence is the thing to watch. Nothing about today's numbers moves. The gate switchover is wired but dormant by construction: it becomes exit-aware the moment the input is enabled, which is exactly what "the certified number is the traded number" has to mean. KNOWN LIMIT, stated rather than hidden: only the AI early-exit route is replayed. The rule-based averaged-vote close (m_threshold_close) depends on every other filter's live vote, which pass 3 does not reproduce, so a position the classic filters would have closed is held to its barrier here. The replay therefore holds LONGER than live and overstates barrier-reached outcomes. Faithful only while the AI is the dominant vote - which is the configuration this is being built for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:25:57 -04:00
bool m_exitReplayReported;
void ReportExitPolicyDivergence(void);
//--- Replays every directional call of this era under the live exit policy. Runs AFTER pass 3, never
//--- inside it: a vote-flip exit for a trade at bar r depends on the decisions at bars r-1, r-2, ...
//--- which pass 3 has not produced yet when it grades r (it walks oldest-to-newest).
void SimulateExitPolicyOutcomes(void);
//--- ExitPolicy() itself is PUBLIC (with the other Warrior_EA.mq5 setters) - it is pushed in
//--- from the EA, not called from inside the class. These two are what it writes.
feat(gates): derive the recall floor instead of configuring it, publish what the configuration can PROVE, and stop an arbitrary member driving live exits Three changes, all from the same principle: measure what is there before aiming at it, and never certify a number you do not trade. 1. THE RECALL FLOOR IS DERIVED, AND IT MOVES BELOW CHANCE. MinRecall=40 was a constant doing a statistical job. Its reference point is the 33.3% recall a zero-skill 3-class model gets on EVERY class, and against that the constant was accidentally calibrated for exactly one sample size: on USDJPY CONV (n_eff 195) 40% is chance + 2.0 SE; on SP500 PAI (n_eff 42) the same 40% is chance + 0.9 SE. One chart was being held to a bar twice as strict as the other, for no reason anyone chose. CollapseRecallFloorPct() computes it per class from that class's own effective sample - EffectiveSampleSize(), so the overlap deflation the rest of the gates use applies here too - as chance - EDGE_MIN_SIGMAS x SE. 26.5% at n_eff 195, 18.8% at n_eff 42. BELOW chance, deliberately, and this is the substantive change rather than the arithmetic. This gate's only job is refusing to call a COLLAPSED model converged. It is not a quality bar; the deploy gate is the quality bar and it is already rigorous (chance + 2 SE on the deflated sample, Sidak over candidate eras, then the cross-instrument pooled certificate). A convergence gate that ALSO demands provably-above-chance recall on all three classes double-counts that job, and it has failed that way twice here: MinRecall=60 blocked every SP500 H1 run in 2026-07, and the 40 that replaced it made Neutral structurally unreachable once first-touch resolution cut Neutral to a 0.65% residue. A floor nothing can reach does not make a funded account safer, it stops the run converging at all. Testing significantly BELOW chance instead catches what a fixed 40 was actually catching - a model that has stopped emitting a class - and cannot become unreachable by construction. It also fixes the direction the old constant scaled: it now widens on a thin OOS window, where low recall genuinely cannot be told from noise, and tightens on a rich one. Today's SP500 PAI (Buy 51 / Sell 18 / Neutral 30) is still correctly blocked on Sell. This also resolves a standing contradiction the code half-admitted at the isBetterEra comment: selection ranks on coverage-weighted PRECISION while convergence gated on RECALL, so a sparse high-precision abstainer - precisely the model that could clear the deploy bar - was blocked by the floor. The era line now PRINTS the derived floor. Anyone comparing these recalls against a remembered "40" is reading the wrong bar. 2. DETECTABILITY: WHAT THIS CONFIGURATION COULD PROVE, BEFORE IT TRAINS. The DEPLOY BAR line states the bar. It never said what reaching it would take, and that is the actionable direction. ReportDetectability() inverts the same identity - the gate passes when edge >= z x sqrt(p(1-p)/n_eff), so certifying an edge d needs n_eff >= z^2 p(1-p)/d^2 independent calls, hence L times as many raw ones - and prints a +2 / +5 / +10pp ladder as required independent calls, raw calls, and share of the OOS window, marking any rung that needs more than the window holds IMPOSSIBLE. Every term is a property of the CONFIGURATION - geometry via break-even, horizon via mean label lifespan, window via oosCutoff - so no amount of training moves any of them. It fires once, at the first healthy sweep, beside ReportFeatureHealth, for the same reason: that is the first moment the bar grid, the measured geometry and the lifespan are real numbers rather than defaults. It gates nothing. This is the EdgeFinder discipline applied to our own gate: establish what the market and the measurement design have to offer, then point the net at it - rather than spending a thousand eras chasing something this OOS window could never certify. 3. AN ARBITRARY MEMBER WAS DRIVING LIVE EXITS AND TRAILING (user-identified). Every ensemble member ran g_LiveAISignedConfidence = SignedAIConfidence(); unconditionally, every tick. Last writer wins. Its consumers are the AI early-exit route (CExpertSignalCustom::LiveSignedConfidence) and TrailingIntelligent - so on a four-model chart an LSTM entry could be closed, and its stop moved, on the Perceptron's opinion alone, decided by scheduling order. Not the vote, not a weighted blend. Now the mean across registered members, matching how the ensemble actually trades: the open decision is the weighted-average vote, and an abstaining member contributes 0 and dilutes exactly as it does there. Members still training read 0, so a half-trained ensemble reads WEAKER rather than louder - the safe direction for an exit trigger. Deployed and paused members are included, which is the opposite of the era barrier's exemption rule and correct for the opposite reason: that one asks who must be waited for, this asks who has an opinion. Latent today and staying that way for now by choice - Min_Vote_Close ships Disabled (101, unreachable on both scales it drives) and TrailingStrategy is off, so live exits are SL/TP only and the certified hold-to-barrier win rate is what actually gets traded. Fixed now precisely because the plan is to enable vote exits once the models are accurate, at which point a scheduling-order exit would be both harmful and very hard to see. STILL OPEN, and needs a decision before vote exits go on: the member gate and the ensemble vote gate both grade hold-to-barrier, so enabling vote exits makes the certified number stop describing the traded one. Warrior_EA.mq5 currently argues barrier models may keep vote exits because "their label IS the vote's own horizon" - that does not hold, since a vote flip at bar 5 of a 64-bar horizon is not the target-before-stop outcome the gate measured. Either grade the OOS call on the real exit rule (first of SL / TP / vote-flip / horizon) through the fill engine, or set HoldToBarrier for ensemble members so the policy cannot drift from the certificate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:15:08 -04:00
double m_lastRecallFloorPct;
//--- Installs an adopted barrier geometry and every side effect that must travel with it: the
//--- derived pair (the one authority), the legacy mode ints, the live-order globals, the .cfg
//--- rewrite, the label cache and the horizon latch.
fix(geometry): the ensemble was training on TWO DIFFERENT TARGETS - propagate the adopted barrier MEASURED 2026-08-17 19:06 on USDJPY, in the fresh run: 19:06:38 LSTM adopting barrier geometry 2:10 ... geometry authority 19:06:40 LSTM triple-barrier labels - stop 2.00 target 10.00, horizon 256 19:06:44 PAI / CONV / HYB break-even 33.3%, mean label lifespan 19.2 bars 19:06:45 LSTM break-even 16.7%, mean label lifespan 81.4 bars One chart, four members, two targets. A "Buy" from LSTM meant "10 ATR before a 2 ATR stop within 256 bars"; a "Buy" from PAI meant "3.21 before 1.61 within 64". The orchestrator averages those votes and the joint gate certifies the average as though they answered one question. And g_DerivedSlAtrMult - which places the LIVE order - is a single global, so the stop actually sent was whichever member wrote last: the same last-writer-wins class of bug as the live-exit confidence. CAUSE, and it is mine. The geometry scan sits at the end of the MI chain, and that chain runs ONCE PER CHART (g_ensembleChartMiReportDone) - whichever member reaches it first measures and the rest skip. Harmless while the scan only PRINTED; 62a719f made it authoritative and turned a skipped report into a skipped DECISION. The indicator tuner already had this doctrine (g_ensembleChartTuneSettings); the geometry had no equivalent. - g_ensembleChartGeomAdopted/Sl/Tp/SlMode/TpMode: the donor publishes its pairing, the siblings adopt it in the MI-skip branch. Ordering is safe by construction - MQL5 is single-threaded per chart and the donor sets g_ensembleChartMiReportDone only after the chain (and so the adoption) returns, so any member taking the skip branch does so strictly afterwards. - ApplyAdoptedGeometry(): the eleven side effects an adopted pairing must carry - derived pair, legacy mode ints, g_Derived* live globals, .cfg rewrite, label cache invalidation, horizon unlatch - in ONE function, because there are now two callers and duplicating them is how the two paths drift. - Guarded on era 0 for the donor's own reason: relabelling a partly trained net moves the target out from under weights already fitted to the old one. STILL OPEN: dead MA handles were not eliminated by cb30360. They now appear at a different site (SP500 19:06:35, during "label prebuild", and on PAI - the member that RAN the sweep), so there is a second handle-churn path I have not found. Recovery works and the sharing diagnosis stands; the trigger is not only the tuner's adopt branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 19:35:45 -04:00
void ApplyAdoptedGeometry(double sl, double tp, int slMode, int tpMode);
//--- ONE-SHOT DETECTABILITY REPORT: how many calls this configuration must fire before the
//--- deploy gate could certify an edge of a given size AT ALL, and what share of the OOS window
//--- that is. Purely a report; it gates nothing.
feat(gates): derive the recall floor instead of configuring it, publish what the configuration can PROVE, and stop an arbitrary member driving live exits Three changes, all from the same principle: measure what is there before aiming at it, and never certify a number you do not trade. 1. THE RECALL FLOOR IS DERIVED, AND IT MOVES BELOW CHANCE. MinRecall=40 was a constant doing a statistical job. Its reference point is the 33.3% recall a zero-skill 3-class model gets on EVERY class, and against that the constant was accidentally calibrated for exactly one sample size: on USDJPY CONV (n_eff 195) 40% is chance + 2.0 SE; on SP500 PAI (n_eff 42) the same 40% is chance + 0.9 SE. One chart was being held to a bar twice as strict as the other, for no reason anyone chose. CollapseRecallFloorPct() computes it per class from that class's own effective sample - EffectiveSampleSize(), so the overlap deflation the rest of the gates use applies here too - as chance - EDGE_MIN_SIGMAS x SE. 26.5% at n_eff 195, 18.8% at n_eff 42. BELOW chance, deliberately, and this is the substantive change rather than the arithmetic. This gate's only job is refusing to call a COLLAPSED model converged. It is not a quality bar; the deploy gate is the quality bar and it is already rigorous (chance + 2 SE on the deflated sample, Sidak over candidate eras, then the cross-instrument pooled certificate). A convergence gate that ALSO demands provably-above-chance recall on all three classes double-counts that job, and it has failed that way twice here: MinRecall=60 blocked every SP500 H1 run in 2026-07, and the 40 that replaced it made Neutral structurally unreachable once first-touch resolution cut Neutral to a 0.65% residue. A floor nothing can reach does not make a funded account safer, it stops the run converging at all. Testing significantly BELOW chance instead catches what a fixed 40 was actually catching - a model that has stopped emitting a class - and cannot become unreachable by construction. It also fixes the direction the old constant scaled: it now widens on a thin OOS window, where low recall genuinely cannot be told from noise, and tightens on a rich one. Today's SP500 PAI (Buy 51 / Sell 18 / Neutral 30) is still correctly blocked on Sell. This also resolves a standing contradiction the code half-admitted at the isBetterEra comment: selection ranks on coverage-weighted PRECISION while convergence gated on RECALL, so a sparse high-precision abstainer - precisely the model that could clear the deploy bar - was blocked by the floor. The era line now PRINTS the derived floor. Anyone comparing these recalls against a remembered "40" is reading the wrong bar. 2. DETECTABILITY: WHAT THIS CONFIGURATION COULD PROVE, BEFORE IT TRAINS. The DEPLOY BAR line states the bar. It never said what reaching it would take, and that is the actionable direction. ReportDetectability() inverts the same identity - the gate passes when edge >= z x sqrt(p(1-p)/n_eff), so certifying an edge d needs n_eff >= z^2 p(1-p)/d^2 independent calls, hence L times as many raw ones - and prints a +2 / +5 / +10pp ladder as required independent calls, raw calls, and share of the OOS window, marking any rung that needs more than the window holds IMPOSSIBLE. Every term is a property of the CONFIGURATION - geometry via break-even, horizon via mean label lifespan, window via oosCutoff - so no amount of training moves any of them. It fires once, at the first healthy sweep, beside ReportFeatureHealth, for the same reason: that is the first moment the bar grid, the measured geometry and the lifespan are real numbers rather than defaults. It gates nothing. This is the EdgeFinder discipline applied to our own gate: establish what the market and the measurement design have to offer, then point the net at it - rather than spending a thousand eras chasing something this OOS window could never certify. 3. AN ARBITRARY MEMBER WAS DRIVING LIVE EXITS AND TRAILING (user-identified). Every ensemble member ran g_LiveAISignedConfidence = SignedAIConfidence(); unconditionally, every tick. Last writer wins. Its consumers are the AI early-exit route (CExpertSignalCustom::LiveSignedConfidence) and TrailingIntelligent - so on a four-model chart an LSTM entry could be closed, and its stop moved, on the Perceptron's opinion alone, decided by scheduling order. Not the vote, not a weighted blend. Now the mean across registered members, matching how the ensemble actually trades: the open decision is the weighted-average vote, and an abstaining member contributes 0 and dilutes exactly as it does there. Members still training read 0, so a half-trained ensemble reads WEAKER rather than louder - the safe direction for an exit trigger. Deployed and paused members are included, which is the opposite of the era barrier's exemption rule and correct for the opposite reason: that one asks who must be waited for, this asks who has an opinion. Latent today and staying that way for now by choice - Min_Vote_Close ships Disabled (101, unreachable on both scales it drives) and TrailingStrategy is off, so live exits are SL/TP only and the certified hold-to-barrier win rate is what actually gets traded. Fixed now precisely because the plan is to enable vote exits once the models are accurate, at which point a scheduling-order exit would be both harmful and very hard to see. STILL OPEN, and needs a decision before vote exits go on: the member gate and the ensemble vote gate both grade hold-to-barrier, so enabling vote exits makes the certified number stop describing the traded one. Warrior_EA.mq5 currently argues barrier models may keep vote exits because "their label IS the vote's own horizon" - that does not hold, since a vote flip at bar 5 of a 64-bar horizon is not the target-before-stop outcome the gate measured. Either grade the OOS call on the real exit rule (first of SL / TP / vote-flip / horizon) through the fill engine, or set HoldToBarrier for ensemble members so the policy cannot drift from the certificate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:15:08 -04:00
void ReportDetectability(int oosBars);
bool m_detectabilityReported;
//--- There is deliberately NO minimum-confidence input here any more, and no member holding one.
//--- That is what makes ONE input genuinely govern both engines. See ConfidenceTier().
bool m_freezePriorCalibration;
//--- THE class-imbalance correction: tau in Menon et al. 0 disables it.
feat(ai): logit-adjusted loss, replacing oversampling and the post-hoc prior Menon et al. 2021 (ICLR), "Long-tail learning via logit adjustment": add tau*log(prior_c) to each class logit inside the training gradient. Softmax CE on adjusted logits is consistent for BALANCED error - the metric checkpoint selection already ranks on - so the loss and the deploy decision finally optimize the same thing. The engine already computed a true softmax + categorical-CE gradient and wrote it over the per-neuron sigmoid delta, so this is an offset added to three logits in the two places that gradient is built (backProp scalar path and backPropOCL). No backend, kernel or DLL change; the forward pass and every inference path are untouched, which is the point - the network learns to absorb the offset, so its raw argmax becomes the balanced-optimal decision with nothing applied at inference. Replaces rather than stacks. Minority replay is disabled while this is on, and the post-hoc inference prior is forced off. Stacking is not a theoretical worry: simulated on the measured 1118/1119/34298 distribution in the weak-signal regime, plain CE collapses to Neutral (33.4% balanced, Buy 0%), replay reaches 48.1%, logit adjustment 50.9% with better balance - and BOTH together score 45.4% with Neutral recall at 0%, worse than either alone. Buda et al. 2018 predicts exactly that. Motivation from the six-chart run: every topology took one direction to ~50% recall and abandoned the other, the direction chosen arbitrarily (the batch-norm control went Buy 1% / Sell 42%, the inverse of the other five). One era in 1,301 cleared the per-class recall floor. Fingerprinted conditionally, so the converged 60.7% models on disk keep their filenames and stay loadable as the fallback. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:05:14 -04:00
double m_logitAdjustTau;
fix(ai): cap logit-adjustment strength to the head's usable logit range tau=1.0 inverted the collapse instead of curing it. The head is SIGMOID, so each output is bounded to [0,1] and the widest logit gap the net can express between two classes is CLASS_LOGIT_SCALE * (1-0) = 6. The offsets are tau*log(prior_c), whose spread on this 30:1 imbalance is 3.42 - so tau=1.0 spent 57% of the ENTIRE expressible range on the prior correction. The network did the only thing available to it: saturate Buy/Sell outputs to 1.0 to overcome a -3.42 training handicap. The offsets are absent at inference, so that surplus made every bar directional. Measured across all five still-training charts: Neutral recall 0%, directional calls on ~100% of bars, win rate 5-7% against a ~6% base rate - no information whatsoever - while balanced accuracy read a flattering 58-64% because two of its three terms sat near 95%. OOS accuracy 6%. Menon et al. assume an unbounded logit head where a 3.42 shift is negligible against the reachable range. It is not negligible here, so the strength is now expressed RELATIVE to the range actually available: tau_eff = min(tau_cfg, LOGIT_ADJUST_MAX_RANGE_FRACTION * SCALE / spread) At 20% that gives tau 0.35 on this data. Deliberately a fraction rather than a tau ceiling: it stays correct if CLASS_LOGIT_SCALE changes, if the head becomes unbounded, or on any symbol whose imbalance differs. The input remains effective below the cap, so dialling it down needs no rebuild. Simulated at a signal strength where the task is genuinely learnable, the precision/recall frontier is monotone: tau 1.0 -> 49.6% call rate at 6.4% precision (base rate 6.1%, i.e. worthless); tau 0.35 -> 2.0% at 15.5%; tau 0.15 -> 0.2% at 33.3%. The capped value lands in the same regime the pre-logit-adjustment run occupied (1-6% of bars at 20-35% win rate). Also logs the measured priors, the spread, and whether the cap bound. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 23:20:07 -04:00
bool m_logitAdjustLogged;
fix: the imbalance correction never ran during the auto-tune search Neutral collapse on all four topologies by era 5 with a 2:6 barrier (recall Buy 0% / Sell 0% / Neutral 100%), and the panel stuck on "measuring...". One root cause, and it was not the barrier. The labels were fine: Buy 25.4% / Sell 22.0% / Neutral 52.5%, which is exactly gambler's ruin for m=2,k=6 (2/8 = 25% per side), with only 0.1% of Neutral coming from the vertical barrier - so the new m*k horizon scaling is right, arguably generous. What was broken: Train()'s era-start block wrapped UpdateClassPriors() in `if(!m_evalMode)`. The auto-tune GA scores every candidate in eval mode, and AutoTuneIndicators ships ON, so on a default configuration EVERY era of the search ran with unmeasured priors. ApplyLogitAdjustment() requires measured priors; without them it calls ClearLogitAdjustment() and returns. So the entire search trained under PLAIN cross-entropy. With a 52.5% majority class the optimum of plain CE is "always predict Neutral", and that is precisely what all four models found. The panel followed: its counters only advance on bars the model CALLED Buy or Sell, so a collapsed model leaves them at zero and the line reads "measuring..." forever. This was latent, not new. It has been true for every auto-tuned run, but it was invisible while the labels were near-balanced - last night's accidental 1:1 barrier gave 43/40/17, where plain CE has no majority to collapse into. Widening the stop to 2*ATR (correctly - 1*ATR is too tight to survive noise) moved Neutral to the majority and exposed it. The guard's stated fear cannot happen. These priors are measured from the LABEL distribution, and the tuner only perturbs indicator periods (MA/RSI/MACD/Ichimoku/AD). The barrier label depends on ATR, SL_Mode and TP_Mode - none of which the search touches - so every candidate sees byte-identical labels and identical priors. There is nothing to contaminate. What the guard actually protected was the .stats write, and that is gated separately: eval candidates never checkpoint and never persist. Also, because this is the THIRD quiet no-op to cost a run in this codebase (after the fictional oversampling log line and the shadow-blend skip): - ApplyLogitAdjustment() now WARNS when it declines to install, instead of silently clearing. A mechanism that cannot announce it is not running is indistinguishable from one that is. - The panel distinguishes "measuring..." (before era 1, nothing scored yet - an honest warm-up) from "no directional calls yet" (eras trained, zero calls - a finding, not a wait). Both builds compile 0 errors / 0 warnings. No retrain forced by this commit itself, but the collapsed models must be discarded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:46:24 -04:00
//--- Latch for the counterpart warning: the correction DECLINING to install. See ApplyLogitAdjustment().
bool m_logitAdjustSkipWarned;
feat(ai): logit-adjusted loss, replacing oversampling and the post-hoc prior Menon et al. 2021 (ICLR), "Long-tail learning via logit adjustment": add tau*log(prior_c) to each class logit inside the training gradient. Softmax CE on adjusted logits is consistent for BALANCED error - the metric checkpoint selection already ranks on - so the loss and the deploy decision finally optimize the same thing. The engine already computed a true softmax + categorical-CE gradient and wrote it over the per-neuron sigmoid delta, so this is an offset added to three logits in the two places that gradient is built (backProp scalar path and backPropOCL). No backend, kernel or DLL change; the forward pass and every inference path are untouched, which is the point - the network learns to absorb the offset, so its raw argmax becomes the balanced-optimal decision with nothing applied at inference. Replaces rather than stacks. Minority replay is disabled while this is on, and the post-hoc inference prior is forced off. Stacking is not a theoretical worry: simulated on the measured 1118/1119/34298 distribution in the weak-signal regime, plain CE collapses to Neutral (33.4% balanced, Buy 0%), replay reaches 48.1%, logit adjustment 50.9% with better balance - and BOTH together score 45.4% with Neutral recall at 0%, worse than either alone. Buda et al. 2018 predicts exactly that. Motivation from the six-chart run: every topology took one direction to ~50% recall and abandoned the other, the direction chosen arbitrarily (the batch-norm control went Buy 1% / Sell 42%, the inverse of the other five). One era in 1,301 cleared the per-class recall floor. Fingerprinted conditionally, so the converged 60.7% models on disk keep their filenames and stay loadable as the fallback. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:05:14 -04:00
//--- True class base rates (natural, un-oversampled), measured from the label distribution each era
//--- (UpdateClassPriors, EMA-blended for stability) and PERSISTED alongside the weights (.stats
//--- sidecar) so live inference - including after a restart, when no training re-runs - calibrates
//--- exactly as training did. 0 = not yet measured => AdjustedSignalFromSoftmax falls back to raw.
double m_priorBuy, m_priorSell, m_priorNeutral;
//--- Per-era OOS "fired" confusion counts under the LIVE decision rule: a directional call is
//--- counted whenever the prior-corrected posterior is non-Neutral, i.e. exactly the bars on
//--- which the deployed EA would cast a directional vote. m_oosBuyFiredHits/m_oosBuyFired = live
//--- Buy precision; likewise Sell.
int m_oosBuyFired, m_oosBuyFiredHits;
//--- The same live-fired population as above, but BUCKETED BY CONFIDENCE TIER (ConfidenceTier(),
//--- 4 buckets quartiled from the head's structural floor).
2026-07-30 11:47:15 -04:00
int m_oosTierFired[4], m_oosTierHits[4];
feat(rank): AI models rank their own confidence tiers from held-out outcomes Closes the caveat 4858507 shipped with: the vote is a confidence percentage, but only to the extent the pattern weights are measured. AI tier weights sat at their designed defaults (25/50/75/100) because AI rows only ever arrive from LIVE journaling, of which a training run produces almost none. AND A STALE-TIER BUG THAT MADE THE EVIDENCE MEANINGLESS. The OOS scan bucketed every scanned bar by ConfidenceTier(), which reads dPrevSignal - and dPrevSignal is assigned in PASS 1 only, never anywhere in the OOS scan. So an entire era's fires were bucketed by one stale, unrelated bar's confidence and landed in a SINGLE tier. That is the "tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0)" symptom recorded on 2026-08-16 and attributed to the calibration clamp. The clamp was real and was fixed then; this is a second, independent cause of the identical output that survived that fix untouched - which is why the log kept reading the same afterwards. Two causes, one symptom. Now ConfidenceTierFor(adjSig): the bar this iteration actually scored. WHY THIS DOES NOT WRITE ROWS TO THE SIGNAL DB, which was the obvious reading of "fill the database during training". The user's own observation is the reason: a classic Pattern_2 is a fixed geometric condition, so its win rate is legitimately accumulated over years, but an AI Pattern_2 means "confidence landed in tier 2" and tier 2 under era 100's weights is a different statement from tier 2 under era 500's. The DB's value is ACCUMULATION, and accumulation is exactly what is wrong here - it would average together models that no longer exist, while colliding with the per-table row cap and mixing measured-on-holdout outcomes into the live ledger's own tables. What the DB actually supplies is a measured win rate per pattern, and pass 3 already computes that on held-out bars, thousands at a time. So the model ranks itself once per era, REPLACING rather than accumulating, which makes the weights describe the current weights by construction. ESTIMATOR. Not WinRateFromCounts(): it returns NO_DATA below 100 raw trades BEFORE shrinking, which here would fire on every tier every era and hand all four the pooled rate - the tiers could never separate and the mechanism would be inert. Shrinkage is the answer to a small sample; a floor in front of it means the shrinkage never runs. Instead: a Beta prior of TIER_PRIOR_EFF_N pseudo-observations centred on the model's pooled holdout rate, counted in EFFECTIVE observations, because overlapping triple-barrier labels mean 800 raw fires can be worth ~12 independent ones. Rounded to the integer, not to the decade NormalizeWinRate() uses, which would collapse the shrunk tiers back into one number. NO SAME-ERA CIRCULARITY, and it falls out of the ordering rather than a guard: weights are computed at the END of era N, so the vote scored during era N was cast with era N-1's weights. The deploy gate never grades a vote whose weights were fitted on the bars it is scoring. Residual leakage remains - the same OOS bars each era under a different model - and is stated in the code rather than papered over. Both DB clobber paths are closed: ApplyPatternWeight() declines once self-ranked, and UpdateSignalsWeights()' filter.Weight() call is guarded by SelfRanked() - guarding only the tiers would have let the hourly ranking pass undo half the self-ranking. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 16:00:32 -04:00
//--- Set by RankTiersFromOos() the first time this model measures its own tiers on held-out
//--- bars. Gates the signal DB out of this filter's pattern weights from then on.
bool m_tiersSelfRanked;
feat(baselines): Alglib forest + linear on the NN's own matrix Every direction verdict so far was measured through one architecture family, so "flat" has two readings that no topology tuning can separate: the net is the wrong learner, or the matrix carries no directional information. Two learners with completely different inductive biases - Alglib's random decision forest and an ordinary least-squares fit - now train on the SAME feature windows (BuildFeatureWindow, the net's own function, so there is no second feature implementation to drift), the SAME labels, the SAME IS/OOS split with both purges, and are scored through the SAME precision-against-always-one-direction comparison and the same Sidak family-wise arithmetic the deploy gate uses. If both also land at chance, the matrix is the limit. Deliberate choices, each of which could have made the comparison a different question wearing this one's name: - LRBuild, not LRBuildZ: the intercept absorbs the class imbalance, and a baseline handicapped by a forced zero intercept would flatter the net for the wrong reason. - Raw call counts in the SE, matching the live gate's known-permissive test rather than correcting it here - both sides must face the same bar. - No threshold sweep on the linear fit: a threshold fitted on the slice being scored is the calibration leak the purged band exists to avoid. - Uniform stride when a cap bites, not the newest N rows, so a score difference cannot be a regime difference. What was dropped is logged. Ships off (Run_Alglib_Baselines = false): it is a measurement, not a trading feature, nothing trades on the answer and no model is saved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 23:45:03 -04:00
//--- One non-NN baseline measurement per run, however many eras follow - see RunBaselineComparison().
bool m_baselineDone;
fix(chart): display now reads era-end SNAPSHOTS - the live cache is wiped mid-era Full-pipeline analysis after "threshold 30, attained often, nothing drawn, still glued to buy". The log falsified the premise before any code did: 21:40:43 swept 4999, 794 voters, drew 491. Strongest 43.0% vs 30.0% 21:42:07 swept 4999, 0 voters, drew 0 21:51:30 swept 4999, 0 voters, drew 0 21:56:30 swept 4999, 922 voters, drew 382. Strongest 44.0% vs 30.0% The arrows WERE drawn - 491 of them, then 382 - and then erased. ONE root cause, three symptoms: every display path read m_arrowSignalCache, which is wiped to sentinel at each era start and only complete again when pass 3 finishes. With eras at ~30s and a sweep at ~17s: * ARROW FLICKER: a sweep landing mid-era found no voters anywhere, and its else-branch deleted the arrow on every voteless bar - erasing the previous sweep's entire output. The chart cycled populated -> blank -> populated; the user kept catching the blank phase. * READOUT GLUE: the newest-cache walk found only sentinel for ~90% of every era and fell through to dPrevSignal - the frozen purge-band edge bar that reads Buy. 659638e fixed which bar was frozen, not the freezing. * VOTER FLAP: 1299 -> 257 -> 1113 across back-to-back sweeps - each saw a different fraction of half-rebuilt caches. THE FIX, structural rather than another patch: 1. Era-end snapshots. RankTiersFromOos() runs at pass-3 completion - the one moment the cache is complete - and now copies it (raw signals, newest LOOKBACK+16 bars) into member-owned snapshot state, unconditionally, BEFORE its early return: an all-Neutral era is a snapshot worth showing, not an absence of one. Raw signals rather than votes, so a tier re-rank between eras reprices them at read time via LiveVoteContribution for free. 2. The sweep (SnapshotVoteAt) and the prospective readout both read snapshots; the readout's fallback chain is live-cache -> snapshot -> dPrevSignal, and the snapshot leg is the one that fires most of the time. 3. NO DATA IS NOT A VERDICT: a den==0 bar no longer deletes - only an actual sub-threshold vote takes an arrow down. This alone ends the wipe half of the flicker even where snapshots are missing (before the first era). 4. Arming moved from an era-counter diff (which fires at era BOUNDARIES, i.e. precisely when caches are about to be wiped) to g_warriorOverlayArmRequest, set by each RankTiersFromOos - "a member's snapshot just got fresher", the only event a redraw can act on. 60s rate limit collapses the four members' burst into one sweep. Classic-only charts arm once at start. 5. Census now reports the direction split - "922 had a voter (610 buy / 312 sell)" - so "the vote leans buy" is checkable from the log instead of inferred from arrow colours. Also visible in the log and worth knowing: the threshold flip-flopped 30 -> 40 -> 30 across the evening's re-inits (census lines at 21:42-21:51 ran at 40), so part of the observed blankness was configuration, not code. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 22:03:20 -04:00
//--- ERA-END SNAPSHOTS of the arrow cache, taken in RankTiersFromOos() at pass-3 completion -
//--- the one moment the cache is complete for the era.
fix(chart): display now reads era-end SNAPSHOTS - the live cache is wiped mid-era Full-pipeline analysis after "threshold 30, attained often, nothing drawn, still glued to buy". The log falsified the premise before any code did: 21:40:43 swept 4999, 794 voters, drew 491. Strongest 43.0% vs 30.0% 21:42:07 swept 4999, 0 voters, drew 0 21:51:30 swept 4999, 0 voters, drew 0 21:56:30 swept 4999, 922 voters, drew 382. Strongest 44.0% vs 30.0% The arrows WERE drawn - 491 of them, then 382 - and then erased. ONE root cause, three symptoms: every display path read m_arrowSignalCache, which is wiped to sentinel at each era start and only complete again when pass 3 finishes. With eras at ~30s and a sweep at ~17s: * ARROW FLICKER: a sweep landing mid-era found no voters anywhere, and its else-branch deleted the arrow on every voteless bar - erasing the previous sweep's entire output. The chart cycled populated -> blank -> populated; the user kept catching the blank phase. * READOUT GLUE: the newest-cache walk found only sentinel for ~90% of every era and fell through to dPrevSignal - the frozen purge-band edge bar that reads Buy. 659638e fixed which bar was frozen, not the freezing. * VOTER FLAP: 1299 -> 257 -> 1113 across back-to-back sweeps - each saw a different fraction of half-rebuilt caches. THE FIX, structural rather than another patch: 1. Era-end snapshots. RankTiersFromOos() runs at pass-3 completion - the one moment the cache is complete - and now copies it (raw signals, newest LOOKBACK+16 bars) into member-owned snapshot state, unconditionally, BEFORE its early return: an all-Neutral era is a snapshot worth showing, not an absence of one. Raw signals rather than votes, so a tier re-rank between eras reprices them at read time via LiveVoteContribution for free. 2. The sweep (SnapshotVoteAt) and the prospective readout both read snapshots; the readout's fallback chain is live-cache -> snapshot -> dPrevSignal, and the snapshot leg is the one that fires most of the time. 3. NO DATA IS NOT A VERDICT: a den==0 bar no longer deletes - only an actual sub-threshold vote takes an arrow down. This alone ends the wipe half of the flicker even where snapshots are missing (before the first era). 4. Arming moved from an era-counter diff (which fires at era BOUNDARIES, i.e. precisely when caches are about to be wiped) to g_warriorOverlayArmRequest, set by each RankTiersFromOos - "a member's snapshot just got fresher", the only event a redraw can act on. 60s rate limit collapses the four members' burst into one sweep. Classic-only charts arm once at start. 5. Census now reports the direction split - "922 had a voter (610 buy / 312 sell)" - so "the vote leans buy" is checkable from the log instead of inferred from arrow colours. Also visible in the log and worth knowing: the threshold flip-flopped 30 -> 40 -> 30 across the evening's re-inits (census lines at 21:42-21:51 ran at 40), so part of the observed blankness was configuration, not code. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 22:03:20 -04:00
double m_overlaySigSnap[];
int m_overlaySnapBars;
double m_prospectiveSigSnap;
feat(hud): per-member neuron lines + a vote label that moves as the nets learn Both 2026-08-19 reports were the same staleness: every source behind the label was an ERA artifact (live cache refills at pass-3 completion, the snapshot copies once per era, dPrevSignal is the frozen purge-band edge bar) - so the readout stepped at era cadence at best, stayed glued to one direction, and lagged the era counter. DisplayInference(): throttled (4s, 1s across an era boundary), SIDE-EFFECT-FREE forward of the current decision bar (window ending on bar 1, same question the live path asks) through the LEARNER net. Batch-norm running stats are bracketed frozen/RESTORED via the new CNet::GetBatchNormFrozen() + CNeuronBatchNormOCL::StatsFrozen() - restore, not unfreeze, because a display tick can land between pass-3 chunks whose whole scan holds them frozen. Writes nothing a trading or training path reads (dPrevSignal, NMS state, tallies, watermarks all untouched; RefreshLatestSignal is not reusable here precisely because it writes all of them). LSTM safe by construction: h/c zeroed per forward. ProspectiveVote() reads the fresh forward as its FIRST source; the era-artifact chain becomes the fallback (meta head, warm-up, window holes). DisplayHudLine(): the reference library's training label, per ensemble member - name, output activations (softmax probs or raw scalar), the decision, its weighted vote (the exact consensus numerator term), era, recent average error, "(trn)" while not vote-capable. Rendered under the vote line in RefreshVoteReadout BEFORE the live-vote defer (member lines are telemetry, not tradable readings), coloured by the member's own direction in muted tones - the vote line's strict green-only-when-it-would-trade rule is untouched. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 08:48:34 -04:00
//--- HUD display state: the last throttled DISPLAY forward's outputs (softmax probabilities for
//--- 3-class heads, the raw scalar in [0] for regression heads), the adjusted signal they
//--- resolve to, and the throttle bookkeeping.
feat(hud): per-member neuron lines + a vote label that moves as the nets learn Both 2026-08-19 reports were the same staleness: every source behind the label was an ERA artifact (live cache refills at pass-3 completion, the snapshot copies once per era, dPrevSignal is the frozen purge-band edge bar) - so the readout stepped at era cadence at best, stayed glued to one direction, and lagged the era counter. DisplayInference(): throttled (4s, 1s across an era boundary), SIDE-EFFECT-FREE forward of the current decision bar (window ending on bar 1, same question the live path asks) through the LEARNER net. Batch-norm running stats are bracketed frozen/RESTORED via the new CNet::GetBatchNormFrozen() + CNeuronBatchNormOCL::StatsFrozen() - restore, not unfreeze, because a display tick can land between pass-3 chunks whose whole scan holds them frozen. Writes nothing a trading or training path reads (dPrevSignal, NMS state, tallies, watermarks all untouched; RefreshLatestSignal is not reusable here precisely because it writes all of them). LSTM safe by construction: h/c zeroed per forward. ProspectiveVote() reads the fresh forward as its FIRST source; the era-artifact chain becomes the fallback (meta head, warm-up, window holes). DisplayHudLine(): the reference library's training label, per ensemble member - name, output activations (softmax probs or raw scalar), the decision, its weighted vote (the exact consensus numerator term), era, recent average error, "(trn)" while not vote-capable. Rendered under the vote line in RefreshVoteReadout BEFORE the live-vote defer (member lines are telemetry, not tradable readings), coloured by the member's own direction in muted tones - the vote line's strict green-only-when-it-would-trade rule is untouched. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 08:48:34 -04:00
double m_dispProbs[3];
double m_dispSignal;
bool m_dispValid;
uint m_dispStamp;
long m_dispEra;
feat(ensemble): per-NN inputs replace the preset selector - the meta head becomes the vote's gate User design (2026-08-19): 'remove the enum menu that selects neural networks... individual inputs for every NN just like classic signals... the META NN should be integrated into the voting decision pipeline when enabled... as a bonus meta labelling is applied to enabled NNs.' - AI_CHOICE is GONE (tombstoned per the stale-.set doctrine). Use_MLP/Use_CONV/Use_LSTM/ Use_CONVLSTM are ordinary bools like the classic votes; the ensemble arithmetic adapts to any subset because the consensus divisor is the enabled capable weight. Two or more enabled = ensemble (|ENS1 token + joint gate, exactly the old AI_HYBRID fingerprints, so existing weight files keep loading); one = the old solo preset; none = classic-only. - Use_MetaLabeling un-couples META from the direction NNs (the old selector made them mutually exclusive). S3 ships: CSignalMETA::LiveMetaGate scores each vote-cleared entry (shared window at bar 1 + proposal descriptor: side, net vote, live geometry, spread/ATR; pattern one-hot ZEROED - ranking, not calibrated probability, documented in the body) and vetoes below the cost-adjusted break-even. Entries only; fail-open everywhere, loudly. - COEXISTENCE HAZARDS closed: VoteCapableWeight()=0 and ProspectiveVote()=false for the meta target - solo-only until today, a trained META would otherwise sit in the consensus divisor as a permanent abstainer and shrink every vote by its module weight. - CERTIFIED == TRADED: the ensemble era verdict replays the identical veto through the same g_warriorMetaGate pointer over its OOS fired bars (bar re-resolved from the row's own time; fail-open counted as fires and reported: 'metaGate: N approved, M vetoed, K unscored'). The overlay deliberately does NOT replay it (veto-filter-in-replay class, calendar-cliff precedent) - documented at the sweep site. Solo charts' own gate does not model the veto - the standing solo-gate caveat, documented at the input. - DB continuity: the pattern/journal DB fingerprint's first slot was (int)AIType; DbLegacyAiSlot() maps every legacy-expressible config to its OLD value (new 2-3 member subsets get 100+bitmask, outside the legacy range) so no existing database re-keys. filterID becomes the enabled roster via one EnabledNNSummary(). - HUD: the meta line shows the gate (armed/(trn), last P vs BE, ok/veto tally); the armed/disarmed announcement fires on state change via one latch (MetaGateArmedNow), not only when an entry happens to be proposed. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 13:01:02 -04:00
//--- META GATE telemetry, read by the HUD's meta line in ChartUI.mqh. The armed latch is
//--- maintained by MetaGateArmedNow() below (any observer may refresh it); the score/counter
//--- fields are written ONLY by CSignalMETA::LiveMetaGate on LIVE queries (barIdx 1) - the
//--- ensemble verdict's historical replays must not inflate the live approve/veto tally.
bool m_metaGateArmed;
double m_metaGateLastP;
double m_metaGateLastBe;
int m_metaGateApproved;
int m_metaGateVetoed;
//--- The ONE readiness test + transition latch for the meta gate: the same test that lets a
//--- direction member's vote count (VoteCapableWeight) - a converged training run on this chart,
//--- or a model loaded for inference-only duty.
feat(ensemble): per-NN inputs replace the preset selector - the meta head becomes the vote's gate User design (2026-08-19): 'remove the enum menu that selects neural networks... individual inputs for every NN just like classic signals... the META NN should be integrated into the voting decision pipeline when enabled... as a bonus meta labelling is applied to enabled NNs.' - AI_CHOICE is GONE (tombstoned per the stale-.set doctrine). Use_MLP/Use_CONV/Use_LSTM/ Use_CONVLSTM are ordinary bools like the classic votes; the ensemble arithmetic adapts to any subset because the consensus divisor is the enabled capable weight. Two or more enabled = ensemble (|ENS1 token + joint gate, exactly the old AI_HYBRID fingerprints, so existing weight files keep loading); one = the old solo preset; none = classic-only. - Use_MetaLabeling un-couples META from the direction NNs (the old selector made them mutually exclusive). S3 ships: CSignalMETA::LiveMetaGate scores each vote-cleared entry (shared window at bar 1 + proposal descriptor: side, net vote, live geometry, spread/ATR; pattern one-hot ZEROED - ranking, not calibrated probability, documented in the body) and vetoes below the cost-adjusted break-even. Entries only; fail-open everywhere, loudly. - COEXISTENCE HAZARDS closed: VoteCapableWeight()=0 and ProspectiveVote()=false for the meta target - solo-only until today, a trained META would otherwise sit in the consensus divisor as a permanent abstainer and shrink every vote by its module weight. - CERTIFIED == TRADED: the ensemble era verdict replays the identical veto through the same g_warriorMetaGate pointer over its OOS fired bars (bar re-resolved from the row's own time; fail-open counted as fires and reported: 'metaGate: N approved, M vetoed, K unscored'). The overlay deliberately does NOT replay it (veto-filter-in-replay class, calendar-cliff precedent) - documented at the sweep site. Solo charts' own gate does not model the veto - the standing solo-gate caveat, documented at the input. - DB continuity: the pattern/journal DB fingerprint's first slot was (int)AIType; DbLegacyAiSlot() maps every legacy-expressible config to its OLD value (new 2-3 member subsets get 100+bitmask, outside the legacy range) so no existing database re-keys. filterID becomes the enabled roster via one EnabledNNSummary(). - HUD: the meta line shows the gate (armed/(trn), last P vs BE, ok/veto tally); the armed/disarmed announcement fires on state change via one latch (MetaGateArmedNow), not only when an entry happens to be proposed. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 13:01:02 -04:00
bool MetaGateArmedNow(void)
{
if(!IsMetaTarget())
return false;
bool armed = (CheckPointer(Net) != POINTER_INVALID
&& (m_trainingComplete || (m_inferenceOnly && m_modelLoadedFromDisk)));
if(armed != m_metaGateArmed)
{
m_metaGateArmed = armed;
Print(ID + (armed
? ": META GATE ARMED - vote-cleared entries are now scored against the"
" cost-adjusted break-even (" + DoubleToString(CostAdjustedBreakEvenPct(), 1) +
"%); below it the entry is vetoed. Scores are the trained context trunk with"
" the pattern term zeroed - ranking quality, not calibrated probability; see"
" LiveMetaGate's header."
: ": META GATE DISARMED - the head is (re)training; entries pass ungated."));
}
return armed;
}
feat(logs): throttle the settled per-era diagnostics - measured 22MB/9.5h of confirmed-working systems Measured from the journal (2026-08-19): the era deep-dive line (~2KB) plus the excursion verdict, tier re-rank, calibration move, barrier hold and selection-regressed note each printed EVERY era for EVERY member - ~940 eras/member/day - long after the systems they watch were confirmed working. Yesterday's file was 1.3GB (70% of it the news-filter calendar spam the sweep fix already removed). VerboseMode returns as an INPUT (demoted 2026-08-01 for the marketplace; that track is dead since the 2026-08-16 pivot) and gains a second job: false throttles each settled per-era print to eras 0-3 plus every TRAIN_LOG_EVERY_ERAS-th (25 ~= one deep-dive per ~15min per member); true restores the per-era firehose, flippable live. Never throttled: anything that marks a CHANGE - new bests, restores + eta decays, plateau stage transitions, deploy approvals, warnings, errors, the label-cache/adoption one-shots, and the combined-vote gate line (the active system's primary telemetry, still every era). Semantic fixes over blanket gating: - barrier hold now ARMS silently and prints only when the hold outlasts the 2-min report interval - a brief hold every era is the design, the long hold is the watchdog case the line exists for; - the ensemble deploy REFUSAL prints immediately when its reason changes (that is a finding), on cadence when unchanged; - the filtered-view census prints when its RESULT moves (drawn count, or strongest vote by >=2pp) and at least every 10th sweep. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 09:36:15 -04:00
//--- Which refusal reason the ensemble deploy verdict printed last (1 = never tradeable,
//--- 2 = joint checkpoint incomplete, 3 = gate not cleared): a CHANGED reason prints
//--- immediately, the same reason repeats only on the TRAIN_LOG_EVERY_ERAS cadence.
int m_lastEnsRefusalKey;
int m_oosSellFired, m_oosSellFiredHits;
//--- Cumulative (compounded, persistent) DIRECTIONAL accuracy = the win-rate of the model's
//--- Buy/Sell calls: of the bars it actually called Buy or Sell, how many matched the true
//--- label.
long m_cumIsCorrect, m_cumIsTotal;
long m_cumOosCorrect, m_cumOosTotal;
//--- Latest live-fired precision (%) and fire count per direction (-1 = n/a), cached at era end for
//--- the status panel/log the same way m_lastBuyRecallPct is (see its comment).
int m_lastBuyFiredPrecPct, m_lastSellFiredPrecPct;
int m_lastBuyFired, m_lastSellFired;
//--- ZigZag repainting embargo, in bars. The stock ZigZag revises its most recent 1-3 legs as
//--- new bars arrive, so a bar's buffer value is trusted only once this many MORE bars have
//--- closed after it. It used to gate the LABELS too, back when the target was the exact
//--- confirmed pivot; the target is now the triple barrier, whose lookahead is its own horizon.
int m_swingConfirmationBars;
//--- Vertical barrier of the triple-barrier label, in bars - see BARRIER_HORIZON_LADDER_COUNT.
//--- Derived once by ComputeBarrierHorizonBars() at the start of the label prebuild and then
//--- held for the run.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
int m_barrierHorizonBars;
fix: refuse invalid SL/TP, fix the unreachable deploy floor, scale the horizon Three defects found by reading the 2026-08-01 training logs, all of which only became visible because the relabel made the numbers mean something. 1. A STALE ENUM TRAINED FOUR MODELS ON THE WRONG TARGET. `OnInit: trade settings snapshot - SL_Mode=1 TP_Mode=-101` -101 was TP_PREV_SWING, deleted from TAKE_PROFIT_MODE on 2026-07-31 in 7eb48f5. MetaTrader does not validate a saved enum input against the enum's current members, so charts saved before that kept the old integer. BarrierMultiples()'s `if(tpMult <= 0.0) tpMult = slMult;` then quietly turned it into a 1:1 barrier, and all four topologies trained ~250 eras against a strategy nobody selected - while the log reported "target 1.00*ATR" as though it were configured. Since the relabel these two inputs ARE the label definition, so this is not a bad trade setting, it is a wrong dataset. ValidateBarrier- Inputs() now refuses to start (INIT_FAILED + Alert + an explicit fix) on any value that is not an enum member. Members are enumerated rather than range-checked because both enums are sparse and carry negative sentinels, so no min/max test can tell a legal value from a deleted one - which is the entire failure mode. The fallback survives as belt-and-braces but now announces itself: a fallback that cannot say it fired is indistinguishable from correct behaviour. 2. THE DEPLOYABILITY FLOOR BECAME MATHEMATICALLY UNREACHABLE. `tradeableOK` required `dirPrecPct >= baseRatePct`, where baseRatePct is Buy+Sell as a share of all bars. At the old exact-pivot target that was ~6%, so "beat the base rate" read as "beat chance" and the test looked sound. Triple-barrier labels put it at ~83%, so the gate now demanded 83% directional precision - impossible by construction. Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing safe to deploy" at a perfectly healthy 43-45% precision, with no checkpoint able to ship however good it got. Replaced with ZERO-SKILL precision, max(Buy,Sell)/allBars: exactly the score of the degenerate always-call-one-direction model this floor exists to reject. Correct at any base rate - ~43% on the current labels, ~3% on the old rare-pivot ones. The era line now prints "(chance N%, edge +Mpp)" beside the selection score, because 44% precision is excellent against a 3% chance level and worthless against a 43% one, and reading the first as the second is what made tonight's run look better than it was. 3. THE HORIZON IGNORED THE BARRIER GEOMETRY. ComputeBarrierHorizonBars() returned the median ZigZag leg, which measures how long a ~1 ATR move takes and says nothing about how long the CONFIGURED barrier needs. First-passage time out of [-m,+k] scales with m*k, so a 1:3 barrier takes ~3x as long as 1:1; the unscaled horizon would have timed out most 1:3 trades and pushed Neutral straight back up, re-creating the imbalance the relabel removes. Now multiplied by slMult*tpMult, calibrated against a real measurement rather than assumed: the accidental 1:1 run resolved at horizon 12 with only 16.7% timeouts, so the swing median is the right scale at m*k=1. Verifiable, not just asserted: the prebuild now counts barriers that ended on the VERTICAL barrier and reports them as a share of Neutral. Neutral conflates "timed out" with "stopped out" and only the first indicts the horizon. Both builds compile 0 errors / 0 warnings. Forces a retrain - correcting TP_Mode re-keys the fingerprint (|TB:1:-101 -> |TB:1:3), which is right: no existing model was trained on the intended target. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:30:49 -04:00
//--- Latch so the invalid-TP fallback in BarrierMultiples() shouts once, not once per labelled bar.
bool m_barrierFallbackWarned;
//--- Did the LAST TripleBarrierLabel() call run out of horizon without either barrier being
//--- touched?
fix: refuse invalid SL/TP, fix the unreachable deploy floor, scale the horizon Three defects found by reading the 2026-08-01 training logs, all of which only became visible because the relabel made the numbers mean something. 1. A STALE ENUM TRAINED FOUR MODELS ON THE WRONG TARGET. `OnInit: trade settings snapshot - SL_Mode=1 TP_Mode=-101` -101 was TP_PREV_SWING, deleted from TAKE_PROFIT_MODE on 2026-07-31 in 7eb48f5. MetaTrader does not validate a saved enum input against the enum's current members, so charts saved before that kept the old integer. BarrierMultiples()'s `if(tpMult <= 0.0) tpMult = slMult;` then quietly turned it into a 1:1 barrier, and all four topologies trained ~250 eras against a strategy nobody selected - while the log reported "target 1.00*ATR" as though it were configured. Since the relabel these two inputs ARE the label definition, so this is not a bad trade setting, it is a wrong dataset. ValidateBarrier- Inputs() now refuses to start (INIT_FAILED + Alert + an explicit fix) on any value that is not an enum member. Members are enumerated rather than range-checked because both enums are sparse and carry negative sentinels, so no min/max test can tell a legal value from a deleted one - which is the entire failure mode. The fallback survives as belt-and-braces but now announces itself: a fallback that cannot say it fired is indistinguishable from correct behaviour. 2. THE DEPLOYABILITY FLOOR BECAME MATHEMATICALLY UNREACHABLE. `tradeableOK` required `dirPrecPct >= baseRatePct`, where baseRatePct is Buy+Sell as a share of all bars. At the old exact-pivot target that was ~6%, so "beat the base rate" read as "beat chance" and the test looked sound. Triple-barrier labels put it at ~83%, so the gate now demanded 83% directional precision - impossible by construction. Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing safe to deploy" at a perfectly healthy 43-45% precision, with no checkpoint able to ship however good it got. Replaced with ZERO-SKILL precision, max(Buy,Sell)/allBars: exactly the score of the degenerate always-call-one-direction model this floor exists to reject. Correct at any base rate - ~43% on the current labels, ~3% on the old rare-pivot ones. The era line now prints "(chance N%, edge +Mpp)" beside the selection score, because 44% precision is excellent against a 3% chance level and worthless against a 43% one, and reading the first as the second is what made tonight's run look better than it was. 3. THE HORIZON IGNORED THE BARRIER GEOMETRY. ComputeBarrierHorizonBars() returned the median ZigZag leg, which measures how long a ~1 ATR move takes and says nothing about how long the CONFIGURED barrier needs. First-passage time out of [-m,+k] scales with m*k, so a 1:3 barrier takes ~3x as long as 1:1; the unscaled horizon would have timed out most 1:3 trades and pushed Neutral straight back up, re-creating the imbalance the relabel removes. Now multiplied by slMult*tpMult, calibrated against a real measurement rather than assumed: the accidental 1:1 run resolved at horizon 12 with only 16.7% timeouts, so the swing median is the right scale at m*k=1. Verifiable, not just asserted: the prebuild now counts barriers that ended on the VERTICAL barrier and reports them as a share of Neutral. Neutral conflates "timed out" with "stopped out" and only the first indicts the horizon. Both builds compile 0 errors / 0 warnings. Forces a retrain - correcting TP_Mode re-keys the fingerprint (|TB:1:-101 -> |TB:1:3), which is right: no existing model was trained on the intended target. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:30:49 -04:00
bool m_lastBarrierTimedOut;
feat(labels): the scheduled close-all is now a vertical barrier in the label walk User report: "I exit everything on Friday close to avoid weekend swap... if the NN training thinks I hold over the weekend it could produce inaccurate results" - it thought exactly that. TripleBarrierLabel walked its full horizon (64 bars, mean lifespan ~18 H4 bars ~ 3 days) straight through the scheduled flat, scoring trades the deployed EA is guaranteed to have closed on Friday 23:45. SQX applies this rule when building strategies; the EA's own labels did not. NextScheduledCloseAll() mirrors CExpertCustom::OnTick's live check exactly (same three inputs, same -1 disabled sentinels, same CLOSE_EVERYDAY semantics, same server clock). The walk stops at the first bar that does not END by the cutoff - OHLC cannot order the tradable fraction of a partial bar, and ties go to the refusal, as everywhere in this file. An unresolved trade at the cutoff times out to Neutral, exactly as live would flatten it. Excursions, the first-passage ladder and the label lifespan truncate with the walk, so the DERIVED geometry is automatically sized to the tradable window - a target the flat rule never lets price reach stops counting as reachable. The prebuild census now splits timeouts: "horizon too short?" vs "ended by the scheduled close-all" - different questions, different fixes. Schedule disabled = no cutoff, exactly like live. Models trained under weekend-blind labels are fitted to a different target; charts with the close-all enabled (the default) should be reset to retrain under the honest labels. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 11:17:14 -04:00
//--- The LAST walk was cut short by the SCHEDULED CLOSE-ALL vertical barrier (the weekend flat
//--- rule) rather than running its full horizon.
feat(labels): the scheduled close-all is now a vertical barrier in the label walk User report: "I exit everything on Friday close to avoid weekend swap... if the NN training thinks I hold over the weekend it could produce inaccurate results" - it thought exactly that. TripleBarrierLabel walked its full horizon (64 bars, mean lifespan ~18 H4 bars ~ 3 days) straight through the scheduled flat, scoring trades the deployed EA is guaranteed to have closed on Friday 23:45. SQX applies this rule when building strategies; the EA's own labels did not. NextScheduledCloseAll() mirrors CExpertCustom::OnTick's live check exactly (same three inputs, same -1 disabled sentinels, same CLOSE_EVERYDAY semantics, same server clock). The walk stops at the first bar that does not END by the cutoff - OHLC cannot order the tradable fraction of a partial bar, and ties go to the refusal, as everywhere in this file. An unresolved trade at the cutoff times out to Neutral, exactly as live would flatten it. Excursions, the first-passage ladder and the label lifespan truncate with the walk, so the DERIVED geometry is automatically sized to the tradable window - a target the flat rule never lets price reach stops counting as reachable. The prebuild census now splits timeouts: "horizon too short?" vs "ended by the scheduled close-all" - different questions, different fixes. Schedule disabled = no cutoff, exactly like live. Models trained under weekend-blind labels are fitted to a different target; charts with the close-all enabled (the default) should be reset to retrain under the honest labels. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 11:17:14 -04:00
bool m_lastLabelWeekendCut;
//--- Did the LAST call find BOTH directions' targets reachable inside the horizon? It is not an
//--- edge case there: it is the whipsaw class, and on SP500 H1 it accounts for nearly all of
//--- Neutral. Published so the prebuild can count it - see m_labelPrebuildBothWonCount for why a
//--- bar that wins in either direction must not be labelled "do not trade".
fix: both-won bars were labelled "do not trade" - resolve by first touch Removing the min-reward:risk raise let the MEASURED geometry come back with the target NEARER than the stop (SP500 H1: target 1.62*ATR at q50 of favourable, stop 3.33*ATR at q75 of adverse). That reopened a branch the code called unreachable: price can reach +target and -target inside one horizon, winning in BOTH directions, and those bars fell through to Neutral. Neutral has only three producers, both-lost is unreachable (you cannot touch -3.33 without crossing -1.62 first, which wins the short), and timeouts logged at 1.0% of Neutral - so ~27% of ALL bars were being handed to the model as the abstain class when a trade either way would have collected its target. The cleanest positives in the sample, labelled "do not trade", while the fitted confidence threshold was being asked to find selectivity in what was left. Resolved by FIRST TOUCH: the target reached earlier is the trade that would have closed first. Same forward window, no extra lookahead. Same-bar ties stay Neutral - OHLC cannot order two touches, and unlike an intrabar stop tie there is no pessimistic side to fall to, so a guess would inject a coin-flip direction into the target. Also: - count both-won and its same-bar tie subset in the prebuild line, so the share is measured rather than inferred from arithmetic on a log line - scope the timeout counter to IS, matching the tally it is reported as a percentage OF; it was incremented over the whole scan and divided by an in-sample denominator - clear m_lastBarrierTimedOut at the top of the walk with the excursions, not at the bottom - the two early returns published the previous bar's verdict - mark the pass-1 label line PROVISIONAL. It prints the enum fallback because geometry can only be derived from excursions that do not exist yet, and it reads exactly like a config change that failed to take effect FORCES RETRAIN. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:35:42 -04:00
bool m_lastBarrierBothWon;
//--- Subset of the above where both targets fell inside the SAME bar, so OHLC cannot say which came
//--- first. Those stay Neutral, for the same reason intrabar ties score as the stop: the file refuses
//--- to order two touches it cannot see the order of.
bool m_lastBarrierBothWonTied;
//--- Did a LONG / a SHORT placed at this bar reach its target before its stop? See
//--- m_oosWinLongTotal for why the deploy gate had to stop using label agreement as its hit
//--- test.
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
bool m_lastWinLong;
bool m_lastWinShort;
diag: is "optimal SL/TP" learnable? Score the features against excursions Proposed direction: train the net to predict entry/SL/TP that maximise return and minimise drawdown, rather than to classify direction. Before rebuilding a head, measure whether the target is learnable at all. That question splits into two that behave nothing alike: HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility clustering is about the most robust regularity in markets. WHICH WAY it goes first (the asymmetry) - direction, which is what every noise-floor verdict in this project has been about. Expectancy comes ONLY from the second. The first buys position sizing and drawdown control - worth having under prop-firm limits, but not an edge: exit management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with expectancy FLAT. Crucially this is NOT already answered. Every MI figure here scored the triple-barrier label, i.e. one specific question at one fixed geometry. A noise-floor result there says nothing about whether excursion MAGNITUDE is learnable - different target, different answer. Four targets, and the verdict is the CONTRAST, printed explicitly because the dangerous misreading of "UP clears" is "we can predict profitable trades": RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that SHOULD clear. Every prior verdict here lacked a control expected to pass; a range target at the floor indicts the measurement, not the market. UP / DOWN - MFE / MAE. ASYMMETRY - up-dn, the only one that can pay. Collected inside the walk the label already does (one max, one min per bar). The early-out when both barriers resolved is GONE: it would have truncated the excursions at whichever bar tripped the last barrier, making the measurement a function of the CURRENT SL/TP - the circularity this is trying to escape. The loop was already bounded by the horizon, so only the average cost moves. Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is fat-tailed and fixed-width bins would put nearly every row in bin 0; it also pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and to the barrier label's ~1.02 instead of confounded by class balance. Two bugs fixed in this code before it ever ran, both of which would have produced a plausible quiet wrong answer rather than an error: - TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the accumulators were reset, so one bar's excursions would be cached under another bar's index. Cleared at the top now, ahead of every return. - An unresolvable bar is still flagged as labelled but carries excursions of exactly 0. Under equal-frequency binning a block of identical zeros drags the lowest cut onto zero and a third of the sample lands in one uninformative bin - a depressed score that reads as "not predictable", a false negative in the direction that would wrongly kill the idea. Rows where both excursions are zero are dropped; price cannot travel zero both ways over a whole horizon. Read-only diagnostic. No topology or label change: no retrain of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 10:22:41 -04:00
//--- Excursions of the bar TripleBarrierLabel() just resolved, in ATR units, published the same way
//--- m_lastBarrierTimedOut is: the walk that finds them is the walk the label already does, so they
//--- cost one max and one min per bar rather than a second pass over history.
double m_lastExcUp; // (maxHigh - entry)/ATR over the horizon, >= 0
double m_lastExcDown; // (entry - minLow)/ATR, >= 0
fix(geometry): a free zero made "never resolve" the winning geometry The CANDIDATE GEOMETRY line shipped in 05f1a53 said per-candidate geometry beats the global pair on every SP500 member at 2-3 sigma. It does not. It said so because a bar that reached neither barrier scored 0 R, and the incumbent's mean is NEGATIVE (-0.07 to -0.21 R). Against a losing baseline a free zero is a win, so the widest candidate always came out ahead - and the reported gain ordered itself by timeout share, not by skill: PAI 95.1% timed out -> +0.189 R (head measured -2.42 sigma, HARMFUL) HYB 73.8% -> +0.182 R (head at chance, +0.68 sigma) CONV 61.8% -> +0.163 R (head measured -2.47 sigma, HARMFUL) LSTM 27.1% -> +0.158 R (head +1.67 sigma) Monotone in the timeout share and inverted against the sigma gate. The acceptance test written when this was built - "the sigma gate predicts LSTM helps and CONV hurts; if the R difference does not reproduce that ordering, something is wrong" - is what caught it. A trade that reaches neither barrier is not worth zero. It is closed at the horizon, which is what the scheduled close-all does live and what SimulateTradeOutcome's timeout path already charges. So mark it there: TripleBarrierLabel now publishes the signed close-to-close travel at the last bar it actually visited (m_termTravelCache, same validity flag as the excursion and ladder caches), and LadderOutcomeR prices a timeout off it instead of returning false. A bar that cannot be evaluated under BOTH pairs is now dropped whole - scoring one leg and defaulting the other is the same bug in a smaller costume. Second defect, same function: CandidateGeometryFor applied neither of the floors the global derivation applies, so on USDJPY it chose stop 2.00 / target 1.00 - a 67% break-even, forbidden by the 1:2 policy floor. c3daded in miniature: a selector optimising its own criterion with no reference to the decision criterion. Both floors now apply, and the ratio is re-checked AFTER the per-leg rung snap, which can lose it. Also: the module weight was an unshrunk pooled win rate. USDJPY ConvLSTM fired 19 times (2.0 effective), won 36.8%, and took module weight 0.37 - 41% of the ensemble's capable weight and the loudest voice on the chart, off two effective observations. It also lifted the computed vote ceiling to 26.3 against a 25 threshold, which is why THRESHOLD UNREACHABLE never printed on a chart whose peak vote is 14 and whose practical ceiling without that member is 18.8. The pooled rate is now shrunk toward the coin-flip rate on the era's own OOS bars over 30 prior-equivalent calls, and the tiers shrink toward the shrunk value rather than the raw one. A member with ~300 effective calls moves by ~0.4pp; the 19-fire member goes 0.37 -> ~0.15. MEASUREMENT ONLY still - no order reads any of this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 11:30:54 -04:00
//--- SIGNED close-to-close travel at the last bar the walk actually visited, in ATR and BEFORE
//--- the spread: what a trade still open when the horizon (or the scheduled close-all) ran out
//--- would be marked at. Positive = price above the entry bar's close. A trade that reaches
//--- neither barrier is NOT worth zero - it is closed at this mark, which is what the live
//--- close-all does and what SimulateTradeOutcome's timeout path already charges.
double m_lastTermTravel;
//--- LABEL LIFESPAN in bars: how long after entry this bar's label became KNOWABLE - the age of
//--- the barrier touch that fixed the outcome, or the full horizon on a timeout. NOT a
//--- diagnostic. See EffectiveSampleSize().
fix(labels): overlapping-label sample correction + horizon cap on the scale ladder Three defects, all surfaced by the 2026-08-17 SP500 H4 run that shipped stop 4.86 / target 9.71 (width 14.57*ATR, horizon 384). 1. EVERY STANDARD ERROR ASSUMED INDEPENDENT SAMPLES. Triple-barrier labels started one per bar overlap by the label's lifespan, so n calls are worth ~n/L independent observations (Lopez de Prado, AFML ch. 4 - sample uniqueness). All three sqrt(p(1-p)/n) sites divided by the RAW count. The tell: the operating point's null-of-the-maximum gate is family-wise and should fire on ~5% of eras under the null. Measured fire rates - PAI 47/73 (64%), ConvLSTM 9/24, LSTM 8/21 (38%), CONV 4/62 (6%). CONV, the only model whose margin distribution admits few bins, sat on the null; the rest cleared a bar that was too low by ~sqrt(L). PAI's deployed threshold consequently alternated between the ENDS of its own range era to era (0.10 -> 0.88 -> 0.86 -> 0.66; coverage 16% <-> 73%). TripleBarrierLabel now records when each label became KNOWABLE - the first winning touch, or both stops, or the timeout - and the prebuild accumulates the mean. EffectiveSampleSize() feeds the operating point, the member deploy gate and the ensemble vote gate. Conservative by construction (n/L is an upper bound on the damage); gates get harder, never easier. 2. THE SCALE LADDER RAN AWAY, again. Horizon scales as swingMedian*sl*tp, and since 4d8cb08 reachability is measured OVER that horizon - so a wider rung buys itself the time that makes it look reachable. Same target -> horizon -> reach -> target loop the excursion window is kept short to avoid; fixing the window confusion reopened it through the other door. It walked 128 -> 256 -> 384 bars and stopped at q90, the widest rung there is, with every rung reading 39-48% against a 20% floor. A floor nothing fails selects nothing. Rungs whose required horizon exceeds BARRIER_HORIZON_MAX are now rejected - the same rule ReportGeometryExpectancyScan already applied. It was printing the shipped pair as CLAMPED and disqualified ('h384!') two lines under the deriver that chose it: two subsystems, one geometry, opposite verdicts. 3. THE RUNG SNAP DESTROYED THE RATIO IT WAS COMPARING. Both legs snapped independently to the coarse first-passage grid, re-rating each candidate: q90 4.86/9.71 -> 5.00/10.00 (2.00), q85 4.07/8.14 -> 5.00/10.00 (IDENTICAL measurement), q75 3.07/6.13 -> 4.00/6.50 (1.63 - a nearer target). So the ladder compared win shares taken at ratios from 1.63 to 2.17 and read the differences as scale. It is why the reach column came out non-monotone in width (q75 48.5% above q90 42.9%). The stop now snaps to its nearest rung in log space and the target follows the ratio off it; the pair actually measured is returned and logged, so a collision reads as a collision. Also: LadderWinShare guarded against the conditional (fractal) geometry path, which fills n from m_fracLegCount while leaving idxList empty - a latent out-of-bounds on a currently-dead path. New log lines: mean label lifespan and effective n on the label-cache line, the required-vs-available horizon per rung, and the grid pair the reconciliation actually measured (its tolerance now scales with the grid skew instead of a flat 5pp). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 01:12:05 -04:00
int m_lastLabelLifespan;
//--- Running mean of the above over every bar the label cache has resolved this process. Rebuilt with
//--- the cache (a horizon change makes every previous measurement answer a different question), and
//--- deliberately a plain mean rather than an EMA: it is a property of the geometry, not a time series.
double m_labelLifespanSum;
long m_labelLifespanCount;
feat(geometry): 1:2 becomes a FLOOR the swing legs may raise, and the scan can no longer undercut it Two coupled changes, both from measurements in today's SP500 H4 log. 1. THE SCAN WAS OVERRIDING THE DERIVER ON THE WRONG OBJECTIVE. At 13:37:47 DeriveBarrierGeometry produced stop 1.21*ATR / target 2.41*ATR - break-even 33.3%. Thirty-seven seconds later the barrier-geometry scan adopted 2:2 - break-even 50.9% - because it carried 0.0143 nats of entry-time information against the configured pair's 0.0075, and cleared its family-wise gate. Information is not expectancy, and the scan says so itself; nothing checked what the adoption did to the operating point. It did this: the fitted thresholds immediately after read 38.8% win vs 50.9% break-even (-12.2pp) and 48.3% vs 50.9% (-2.6pp), where the earlier model on this instrument at a 1:2 geometry had fitted +1.8pp. The deriver applies the ratio as user RISK POLICY; a scan that can crown 1:1 makes two subsystems disagree about one geometry - the same split this file already fixed once for the clamped-horizon rule. The scan now enrols and crowns only pairings at or above the floor; sub-floor pairs are still scored and printed (marked 'r') so the choice stays auditable. This is NOT the min-RR rule removed on 2026-08-09 - that one guarded a rejection filter that no longer exists. 2. THE RATIO IS A FLOOR, NOT A CAP (user: "the ratio of 1:2 is a minimum that I want, but it should not cap to that if the average zigzag moves gives more room"). BARRIER_TARGET_RR -> BARRIER_TARGET_RR_MIN. ComputeBarrierHorizonBars already scanned ZigZag pivots for leg DURATION; it now harvests leg RANGE in the same pass - two properties of one object, so the horizon and the target describe the same legs instead of two windows. The per-rung ratio is the floor raised toward median-leg/stop, snapped DOWN to a coarse ladder (2.0/2.5/3.0/4.0/5.0). The ladder is coarse on purpose: PooledGate pools only instruments whose structural break-even matches, and continuous per-instrument ratios would never match and would silently empty the pool. A leg is the right yardstick precisely because it owes NOTHING to the barrier - sizing a target off travel measured over the barrier's own horizon is the circular loop that ran EURUSD/USDCAD away to 14-31*ATR in 2026-08-07. The raise stays bounded by the three tests already in the ladder: reachability, the horizon ceiling (first-passage time grows with stop x target), and the cost fraction. Consequential fixes: the reachability floor was a macro keyed to the fixed ratio and is now BarrierMinReachPct(rr) evaluated per rung (a raised ratio has a lower break-even, so a fixed floor would be the wrong strictness); the detectability break-even likewise; PooledGate now writes and matches the ACTUAL ratio (TargetRR()) rather than the floor. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 16:16:33 -04:00
//--- The ratio the derivation actually landed on (target/stop). At or above BARRIER_TARGET_RR_MIN
//--- by construction; PooledGate keys poolability on it, since it IS the structural break-even.
double TargetRR(void) const
{
return (m_derivedSlMult > 0.0) ? (m_derivedTpMult / m_derivedSlMult) : BARRIER_TARGET_RR_MIN;
}
//--- DERIVED barrier multiples, in ATR units, taken from the measured excursion distribution
//--- rather than from an enum. Zero means "not derived yet" and BarrierMultiples() falls back to
//--- the mode constants.
feat: derive the ATR multiples from measured excursions - no hardcoded geometry The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in 3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of eleven guesses is not deriving anything. WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It ranks pairings by how predictable their OUTCOME is - a question about direction. The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x. Hence the scan failing its own gate on every run, and its "winner" wandering 2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is strongly measurable, so derive the geometry from that instead. stop = q25 of measured ADVERSE travel (ordinary noise does not reach it) target = q50 of measured FAVOURABLE travel (reached ~half the time, by construction, inside the horizon) Continuous, in ATR units, superseding the enum multiples. Reachability ("target on X% of bars, stop on Y%") and the implied break-even are printed so the choice is auditable rather than trusted. FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the horizon with the target (first-passage time grows with the band) and the excursions are measured OVER the horizon, so target -> horizon -> excursions -> target is a real loop - deriving once sizes the target from travel measured under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at 3 passes, and says so if it does not settle. Does NOT create expectancy, and the log says as much: chance precision equals break-even at every geometry (m/(m+k) on both sides). It buys a target the market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio forces a target the market rarely reaches, it WARNS rather than overriding - the ratio is the user's risk policy, so the honest move is to state its cost. That is the collision that once rejected 100% of setups. Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg files written earlier today still load (their length guard finds no doubles) and a model that carries them was trained on them and never re-derives. Also fixes a message from e5ceed6 that claimed "this model resumed from disk" unconditionally - it printed above a "seeding era 0" line on a brand-new model, because the branch fires whenever the cache is not built, which is equally true before a fresh model's first prebuild. A diagnostic that misreports its own trigger is worse than one that says nothing: it gets quoted back as evidence. FORCES A FULL RETRAIN (labels change). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 12:06:25 -04:00
double m_derivedSlMult;
double m_derivedTpMult;
bool m_geometryDerived;
//--- DIRECTIONAL CONFIDENCE THRESHOLD - see DIR_CONF_THRESHOLD_BINS for the rationale. Refitted
//--- at the end of every pass 2 from that era's own IS margins, because the margin distribution
//--- moves with the weights.
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
double m_dirConfThreshold;
//--- The value that belongs to the CHECKPOINTED weights.
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
double m_bestDirConfThreshold;
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- Margin histogram for the fit, rebuilt each era from the CALIBRATION slice (see
//--- DIR_CONF_CALIB_PCT_OF_IS).
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
long m_dirConfBinCalls[DIR_CONF_THRESHOLD_BINS];
long m_dirConfBinHits[DIR_CONF_THRESHOLD_BINS];
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
long m_dirConfPrimaryBars; // denominator for coverage: every calibration bar scored
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//--- one-shot so the "histogram too sparse" explanation is stated once per run, not once per era
bool m_dirConfSparseWarned;
feat: derive the ATR multiples from measured excursions - no hardcoded geometry The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in 3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of eleven guesses is not deriving anything. WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It ranks pairings by how predictable their OUTCOME is - a question about direction. The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x. Hence the scan failing its own gate on every run, and its "winner" wandering 2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is strongly measurable, so derive the geometry from that instead. stop = q25 of measured ADVERSE travel (ordinary noise does not reach it) target = q50 of measured FAVOURABLE travel (reached ~half the time, by construction, inside the horizon) Continuous, in ATR units, superseding the enum multiples. Reachability ("target on X% of bars, stop on Y%") and the implied break-even are printed so the choice is auditable rather than trusted. FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the horizon with the target (first-passage time grows with the band) and the excursions are measured OVER the horizon, so target -> horizon -> excursions -> target is a real loop - deriving once sizes the target from travel measured under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at 3 passes, and says so if it does not settle. Does NOT create expectancy, and the log says as much: chance precision equals break-even at every geometry (m/(m+k) on both sides). It buys a target the market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio forces a target the market rarely reaches, it WARNS rather than overriding - the ratio is the user's risk policy, so the honest move is to state its cost. That is the collision that once rejected 100% of setups. Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg files written earlier today still load (their length guard finds no doubles) and a model that carries them was trained on them and never re-derives. Also fixes a message from e5ceed6 that claimed "this model resumed from disk" unconditionally - it printed above a "seeding era 0" line on a brand-new model, because the branch fires whenever the cache is not built, which is equally true before a fresh model's first prebuild. A diagnostic that misreports its own trigger is worse than one that says nothing: it gets quoted back as evidence. FORCES A FULL RETRAIN (labels change). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 12:06:25 -04:00
int m_geometryDerivePasses; // fixed-point iteration counter, capped
fix: a restart no longer loses the measured geometry or the training window Terminal restart, 22:25: all four resumed models sat on empty windows with enum 2:6 barriers. Three interlocking causes, all visible in one log excerpt: 1) THE PRE-SCAN WINDOW WAS SIZED BY THE SAVED WATERMARK. A resumed model's dtStudied sits at its last studied bar, so Bars(dtStudied, now) ~ 0 and the resumed-model MI pre-scan built a zero-bar "complete" label cache - logged as "Buy: 0 | Sell: 0 | Neutral: 0". Train()'s own era start RESETS dtStudied to the training-window rule before computing its window; the pre-scan did not. The rule is now factored into TrainWindowStart() and both use it. The scan also refuses to arm before SERIES_SYNCHRONIZED (it ran in the same second as OnInit), and deployed models keep their watermark - for them it gates inference recency, not a training window. 2) THE HORIZON LATCHED ON AN INDICATOR WARM-UP. ComputeBarrierHorizonBars ran against a ZigZag with 0 calculated legs, fell back, and EnsureBarrierHorizon latched fallback(32) x slMult x tpMult = 384 for the process lifetime. A leg-starved horizon is now PROVISIONAL: re-resolved on the next rebuild, the label cache wiped if it moved (labels from two horizons answer different questions), and the geometry deriver refuses to run from it - a pair derived over a warm-up window would get PINNED. 3) THE DERIVED GEOMETRY WAS NEVER PERSISTED. The .cfg is written at model creation and at weights-reset - both BEFORE era 0 derives - so the measured pair lived only in memory: every restart read back zeros, adopted nothing, fell back to the enum barriers, and the era-0-only gate meant a resumed model could NEVER re-derive. A full day of training on 3.33/1.62 resumed as 2:6. Now: the settled pair is pinned to the .cfg the moment derivation completes (one-shot, atomic write), and the derive gate accepts any model with no pinned pair, not just era 0 - mid-run stability is carried by m_geometryDerived itself, which never allows a second derivation. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 22:40:43 -04:00
//--- The last ComputeBarrierHorizonBars() ran with FEWER confirmed ZigZag legs than the median
//--- needs, so the horizon it returned is the fallback, not a measurement.
fix: a restart no longer loses the measured geometry or the training window Terminal restart, 22:25: all four resumed models sat on empty windows with enum 2:6 barriers. Three interlocking causes, all visible in one log excerpt: 1) THE PRE-SCAN WINDOW WAS SIZED BY THE SAVED WATERMARK. A resumed model's dtStudied sits at its last studied bar, so Bars(dtStudied, now) ~ 0 and the resumed-model MI pre-scan built a zero-bar "complete" label cache - logged as "Buy: 0 | Sell: 0 | Neutral: 0". Train()'s own era start RESETS dtStudied to the training-window rule before computing its window; the pre-scan did not. The rule is now factored into TrainWindowStart() and both use it. The scan also refuses to arm before SERIES_SYNCHRONIZED (it ran in the same second as OnInit), and deployed models keep their watermark - for them it gates inference recency, not a training window. 2) THE HORIZON LATCHED ON AN INDICATOR WARM-UP. ComputeBarrierHorizonBars ran against a ZigZag with 0 calculated legs, fell back, and EnsureBarrierHorizon latched fallback(32) x slMult x tpMult = 384 for the process lifetime. A leg-starved horizon is now PROVISIONAL: re-resolved on the next rebuild, the label cache wiped if it moved (labels from two horizons answer different questions), and the geometry deriver refuses to run from it - a pair derived over a warm-up window would get PINNED. 3) THE DERIVED GEOMETRY WAS NEVER PERSISTED. The .cfg is written at model creation and at weights-reset - both BEFORE era 0 derives - so the measured pair lived only in memory: every restart read back zeros, adopted nothing, fell back to the enum barriers, and the era-0-only gate meant a resumed model could NEVER re-derive. A full day of training on 3.33/1.62 resumed as 2:6. Now: the settled pair is pinned to the .cfg the moment derivation completes (one-shot, atomic write), and the derive gate accepts any model with no pinned pair, not just era 0 - mid-run stability is carried by m_geometryDerived itself, which never allows a second derivation. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 22:40:43 -04:00
bool m_barrierHorizonLegStarved;
bool m_horizonStarvedWarned; // one-shot: the starved path can retry every call
//--- Has THIS process written the derived geometry into the .cfg? Set by the post-derivation
//--- save, and also by the adoption path (the pair is already on disk there).
fix: a restart no longer loses the measured geometry or the training window Terminal restart, 22:25: all four resumed models sat on empty windows with enum 2:6 barriers. Three interlocking causes, all visible in one log excerpt: 1) THE PRE-SCAN WINDOW WAS SIZED BY THE SAVED WATERMARK. A resumed model's dtStudied sits at its last studied bar, so Bars(dtStudied, now) ~ 0 and the resumed-model MI pre-scan built a zero-bar "complete" label cache - logged as "Buy: 0 | Sell: 0 | Neutral: 0". Train()'s own era start RESETS dtStudied to the training-window rule before computing its window; the pre-scan did not. The rule is now factored into TrainWindowStart() and both use it. The scan also refuses to arm before SERIES_SYNCHRONIZED (it ran in the same second as OnInit), and deployed models keep their watermark - for them it gates inference recency, not a training window. 2) THE HORIZON LATCHED ON AN INDICATOR WARM-UP. ComputeBarrierHorizonBars ran against a ZigZag with 0 calculated legs, fell back, and EnsureBarrierHorizon latched fallback(32) x slMult x tpMult = 384 for the process lifetime. A leg-starved horizon is now PROVISIONAL: re-resolved on the next rebuild, the label cache wiped if it moved (labels from two horizons answer different questions), and the geometry deriver refuses to run from it - a pair derived over a warm-up window would get PINNED. 3) THE DERIVED GEOMETRY WAS NEVER PERSISTED. The .cfg is written at model creation and at weights-reset - both BEFORE era 0 derives - so the measured pair lived only in memory: every restart read back zeros, adopted nothing, fell back to the enum barriers, and the era-0-only gate meant a resumed model could NEVER re-derive. A full day of training on 3.33/1.62 resumed as 2:6. Now: the settled pair is pinned to the .cfg the moment derivation completes (one-shot, atomic write), and the derive gate accepts any model with no pinned pair, not just era 0 - mid-run stability is carried by m_geometryDerived itself, which never allows a second derivation. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 22:40:43 -04:00
bool m_geometryCfgSaved;
//--- ADOPTED-GEOMETRY LATCH. True once ReportBarrierGeometryScan has crowned a pairing that
//--- cleared its family-wise null. The scan wins, and this latch is how. So the scan's decision
//--- was inert, and had it not been it would have been overwritten by the next derive pass
//--- anyway.
fix(consistency): one geometry authority, one exit authority, and the MI screen finally gets a veto Consistency pass before a fresh deployment. Three places where two systems were choosing the same thing and one of them silently lost. 1. THE GEOMETRY SCAN'S DECISION WAS INERT - measured, not suspected. USDJPY, 2026-08-17: 14:24:12.844 adopting barrier geometry 2:8 ... Relabelling and training on it. 14:24:12.979 triple-barrier labels - stop 1.61*ATR, target 3.21*ATR ... this is what trains It adopted 2:8 and trained on 1.61:3.21. ReportBarrierGeometryScan wrote only m_sl_mode/m_tp_mode, and BarrierMultiples ranks the DERIVED pair ABOVE those ints - so on any model carrying a derived pair (every model with a .cfg, including a fresh one whose weights are gone but whose sidecar survived) the adoption changed nothing. Worse, had it changed something it would have been undone immediately: the adoption sets m_labelCachePrebuilt = false, and that prebuild re-runs DeriveBarrierGeometry at era 0, which overwrites m_derivedSl/TpMult from the excursion quantiles. ONE AUTHORITY: the derived pair, because it is what the labels read, what the deploy gate certifies, what g_Derived*AtrMult places on the live order, and what the .cfg pins across restarts. The scan now writes THAT (floored by MIN_SL_ATR_MULTIPLIER, the same floor DeriveBarrierGeometry applies so the live stop can never be wider than the labelled one), republishes to the bridge immediately rather than at the next era end, and forces the sidecar to be rewritten. m_geometryAdopted latches it so the derive pass the adoption itself triggers cannot overwrite it. The scan outranks the derive for an evidential reason, not an architectural one: its winner cleared a permutation test against the null of the MAXIMUM over every eligible pairing, and it scores the incumbent derived pair as a peer in that same field. The derive is a descriptive quantile read with no significance test attached. BEHAVIOURAL CHANGE, and the reason to flag it before a fresh test: barrier geometry will now actually move when the scan says so. Until today it never did. 2. ONE EXIT AUTHORITY, tied to whose certificate the trade was placed under. CheckClosePosition had two routes. The AI early-exit reads the AI vote undiluted and is exactly what the new exit replay reproduces. The blended route thresholds m_direction, the average over EVERY filter including classic ones whose live votes pass 3 never computes - so it can close a position the certificate never modelled, and no replay can ever check it. When g_DerivedSlAtrMult > 0 the AI's measured geometry is on the order, which means the deploy gate's certificate is the reason the trade exists. In that state the AI now governs the exit and the blended route is suppressed. Classic-only configurations are untouched: there the blended route is the only exit opinion and stays exactly as it was. Nothing moves at the shipped defaults either way (Min_Vote_Close = Disabled). 3. EDGEFINDER, SECOND HALF: THE MEASUREMENT NOW STEERS. The MI suite has always printed its verdicts and then trained the direction target regardless of what they said. That gap IS the difference between this and the EdgeFinder discipline: measure what the market offers, THEN aim. m_dirEvidence is set when EITHER the feature/label mutual information OR the normalised excursion asymmetry clears its block-permuted null - an OR, because the two look for the same thing by different routes and requiring both would reject on the weaker of two independent measurements. Normalised asymmetry specifically, never the raw one, which is the volatility confound. Deploy - solo AND ensemble - now requires it. A run without it still trains, and keeps its checkpoint: the research value is real and the measurement can be wrong. It simply may not go live. Reported separately from the statistical gate because the remedy is different: a failed selection test says train differently, this says look somewhere else. Excursion SIZE keeps clearing where direction does not, and that is a risk-control head rather than an entry signal. For the ensemble the check is per-chart by construction - the MI suite runs once and shares its outcome across members - which is the honest treatment: four models finding nothing between them is not four chances at an edge, it is four fits to the same absent information. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:43:20 -04:00
bool m_geometryAdopted;
//--- MEASURED DIRECTIONAL EVIDENCE, and the one thing that makes the MI suite a SCREEN rather
//--- than a commentary.
fix(consistency): one geometry authority, one exit authority, and the MI screen finally gets a veto Consistency pass before a fresh deployment. Three places where two systems were choosing the same thing and one of them silently lost. 1. THE GEOMETRY SCAN'S DECISION WAS INERT - measured, not suspected. USDJPY, 2026-08-17: 14:24:12.844 adopting barrier geometry 2:8 ... Relabelling and training on it. 14:24:12.979 triple-barrier labels - stop 1.61*ATR, target 3.21*ATR ... this is what trains It adopted 2:8 and trained on 1.61:3.21. ReportBarrierGeometryScan wrote only m_sl_mode/m_tp_mode, and BarrierMultiples ranks the DERIVED pair ABOVE those ints - so on any model carrying a derived pair (every model with a .cfg, including a fresh one whose weights are gone but whose sidecar survived) the adoption changed nothing. Worse, had it changed something it would have been undone immediately: the adoption sets m_labelCachePrebuilt = false, and that prebuild re-runs DeriveBarrierGeometry at era 0, which overwrites m_derivedSl/TpMult from the excursion quantiles. ONE AUTHORITY: the derived pair, because it is what the labels read, what the deploy gate certifies, what g_Derived*AtrMult places on the live order, and what the .cfg pins across restarts. The scan now writes THAT (floored by MIN_SL_ATR_MULTIPLIER, the same floor DeriveBarrierGeometry applies so the live stop can never be wider than the labelled one), republishes to the bridge immediately rather than at the next era end, and forces the sidecar to be rewritten. m_geometryAdopted latches it so the derive pass the adoption itself triggers cannot overwrite it. The scan outranks the derive for an evidential reason, not an architectural one: its winner cleared a permutation test against the null of the MAXIMUM over every eligible pairing, and it scores the incumbent derived pair as a peer in that same field. The derive is a descriptive quantile read with no significance test attached. BEHAVIOURAL CHANGE, and the reason to flag it before a fresh test: barrier geometry will now actually move when the scan says so. Until today it never did. 2. ONE EXIT AUTHORITY, tied to whose certificate the trade was placed under. CheckClosePosition had two routes. The AI early-exit reads the AI vote undiluted and is exactly what the new exit replay reproduces. The blended route thresholds m_direction, the average over EVERY filter including classic ones whose live votes pass 3 never computes - so it can close a position the certificate never modelled, and no replay can ever check it. When g_DerivedSlAtrMult > 0 the AI's measured geometry is on the order, which means the deploy gate's certificate is the reason the trade exists. In that state the AI now governs the exit and the blended route is suppressed. Classic-only configurations are untouched: there the blended route is the only exit opinion and stays exactly as it was. Nothing moves at the shipped defaults either way (Min_Vote_Close = Disabled). 3. EDGEFINDER, SECOND HALF: THE MEASUREMENT NOW STEERS. The MI suite has always printed its verdicts and then trained the direction target regardless of what they said. That gap IS the difference between this and the EdgeFinder discipline: measure what the market offers, THEN aim. m_dirEvidence is set when EITHER the feature/label mutual information OR the normalised excursion asymmetry clears its block-permuted null - an OR, because the two look for the same thing by different routes and requiring both would reject on the weaker of two independent measurements. Normalised asymmetry specifically, never the raw one, which is the volatility confound. Deploy - solo AND ensemble - now requires it. A run without it still trains, and keeps its checkpoint: the research value is real and the measurement can be wrong. It simply may not go live. Reported separately from the statistical gate because the remedy is different: a failed selection test says train differently, this says look somewhere else. Excursion SIZE keeps clearing where direction does not, and that is a risk-control head rather than an entry signal. For the ensemble the check is per-chart by construction - the MI suite runs once and shares its outcome across members - which is the honest treatment: four models finding nothing between them is not four chances at an edge, it is four fits to the same absent information. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:43:20 -04:00
bool m_dirEvidence;
string m_dirEvidenceWhy;
fix: excursion window must not depend on the barrier it sizes DIRECTION IS NOT THERE, and this run is what establishes it. Three symbols: raw ASYMMETRY clears on all three (p=0.0199 / 0.0050 / 0.0050) norm ASYMMETRY collapses on all three (p=0.3433 / 0.5075 / 0.2736), USDCAD landing BELOW its own null RANGE control strengthens to 3-5x its null everywhere Divide sigma out and the apparent directional signal vanishes entirely. What cleared was volatility leaking through an unnormalised difference. Note this would have passed any replication test: three instruments at p=0.005 is exactly the evidence one would accept before committing to a rebuild, and the confound reproduces perfectly. Replication was never going to catch it - only the normalisation could. Two defects of mine, both surfaced by the same run. 1. THE GEOMETRY DERIVATION WAS DIVERGING, NOT CONVERGING. It produced a 14.57*ATR stop and a 29.14*ATR target that only 5.7% of bars ever reach. Excursions were measured over the barrier horizon; the horizon scales with the target; the target is a quantile of the excursions - so target -> horizon -> excursions -> target ran away, and "settled" only because the horizon ladder caps at 384 bars. A saturated runaway, which the iteration guard could not catch because it watches for OSCILLATION. Fixed at the root: excursions now accumulate only over m_swingMedianBars - the UNSCALED median ZigZag leg, a property of the instrument that owes nothing to the barrier. The barrier walk still runs the full horizon, because that is how long the trade is held; only the MEASUREMENT used to size the barrier is confined to a geometry-independent window. (The Min_Risk_Reward_Ratio warning fired correctly and is what flagged it - the diagnostic worked while the derivation behind it did not.) 2. THE CONFOUND VERDICT WAS UNREACHABLE. `sizeCleared && !asymCleared` was tested first and is true whenever size clears - i.e. always - so the branch that NAMES the volatility confound never printed; all three symbols showed the generic size-not-direction message instead. Verdict chain rewritten with the specific case first, and the dangling elses my first patch introduced removed. FORCES A FULL RETRAIN (the excursion window changes every derived barrier). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 13:57:23 -04:00
//--- Median confirmed ZigZag leg in bars, UNSCALED by the barrier. The window excursions are measured
//--- over, kept independent of the geometry so sizing the geometry from them cannot feed back.
int m_swingMedianBars;
feat(geometry): 1:2 becomes a FLOOR the swing legs may raise, and the scan can no longer undercut it Two coupled changes, both from measurements in today's SP500 H4 log. 1. THE SCAN WAS OVERRIDING THE DERIVER ON THE WRONG OBJECTIVE. At 13:37:47 DeriveBarrierGeometry produced stop 1.21*ATR / target 2.41*ATR - break-even 33.3%. Thirty-seven seconds later the barrier-geometry scan adopted 2:2 - break-even 50.9% - because it carried 0.0143 nats of entry-time information against the configured pair's 0.0075, and cleared its family-wise gate. Information is not expectancy, and the scan says so itself; nothing checked what the adoption did to the operating point. It did this: the fitted thresholds immediately after read 38.8% win vs 50.9% break-even (-12.2pp) and 48.3% vs 50.9% (-2.6pp), where the earlier model on this instrument at a 1:2 geometry had fitted +1.8pp. The deriver applies the ratio as user RISK POLICY; a scan that can crown 1:1 makes two subsystems disagree about one geometry - the same split this file already fixed once for the clamped-horizon rule. The scan now enrols and crowns only pairings at or above the floor; sub-floor pairs are still scored and printed (marked 'r') so the choice stays auditable. This is NOT the min-RR rule removed on 2026-08-09 - that one guarded a rejection filter that no longer exists. 2. THE RATIO IS A FLOOR, NOT A CAP (user: "the ratio of 1:2 is a minimum that I want, but it should not cap to that if the average zigzag moves gives more room"). BARRIER_TARGET_RR -> BARRIER_TARGET_RR_MIN. ComputeBarrierHorizonBars already scanned ZigZag pivots for leg DURATION; it now harvests leg RANGE in the same pass - two properties of one object, so the horizon and the target describe the same legs instead of two windows. The per-rung ratio is the floor raised toward median-leg/stop, snapped DOWN to a coarse ladder (2.0/2.5/3.0/4.0/5.0). The ladder is coarse on purpose: PooledGate pools only instruments whose structural break-even matches, and continuous per-instrument ratios would never match and would silently empty the pool. A leg is the right yardstick precisely because it owes NOTHING to the barrier - sizing a target off travel measured over the barrier's own horizon is the circular loop that ran EURUSD/USDCAD away to 14-31*ATR in 2026-08-07. The raise stays bounded by the three tests already in the ladder: reachability, the horizon ceiling (first-passage time grows with stop x target), and the cost fraction. Consequential fixes: the reachability floor was a macro keyed to the fixed ratio and is now BarrierMinReachPct(rr) evaluated per rung (a raised ratio has a lower break-even, so a fixed floor would be the wrong strictness); the detectability break-even likewise; PooledGate now writes and matches the ACTUAL ratio (TargetRR()) rather than the floor. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 16:16:33 -04:00
//--- Median confirmed ZigZag leg RANGE, in ATR units - the price twin of m_swingMedianBars, and
//--- measured in the same pivot scan (ComputeBarrierHorizonBars) so the two describe the same
//--- legs.
feat(geometry): 1:2 becomes a FLOOR the swing legs may raise, and the scan can no longer undercut it Two coupled changes, both from measurements in today's SP500 H4 log. 1. THE SCAN WAS OVERRIDING THE DERIVER ON THE WRONG OBJECTIVE. At 13:37:47 DeriveBarrierGeometry produced stop 1.21*ATR / target 2.41*ATR - break-even 33.3%. Thirty-seven seconds later the barrier-geometry scan adopted 2:2 - break-even 50.9% - because it carried 0.0143 nats of entry-time information against the configured pair's 0.0075, and cleared its family-wise gate. Information is not expectancy, and the scan says so itself; nothing checked what the adoption did to the operating point. It did this: the fitted thresholds immediately after read 38.8% win vs 50.9% break-even (-12.2pp) and 48.3% vs 50.9% (-2.6pp), where the earlier model on this instrument at a 1:2 geometry had fitted +1.8pp. The deriver applies the ratio as user RISK POLICY; a scan that can crown 1:1 makes two subsystems disagree about one geometry - the same split this file already fixed once for the clamped-horizon rule. The scan now enrols and crowns only pairings at or above the floor; sub-floor pairs are still scored and printed (marked 'r') so the choice stays auditable. This is NOT the min-RR rule removed on 2026-08-09 - that one guarded a rejection filter that no longer exists. 2. THE RATIO IS A FLOOR, NOT A CAP (user: "the ratio of 1:2 is a minimum that I want, but it should not cap to that if the average zigzag moves gives more room"). BARRIER_TARGET_RR -> BARRIER_TARGET_RR_MIN. ComputeBarrierHorizonBars already scanned ZigZag pivots for leg DURATION; it now harvests leg RANGE in the same pass - two properties of one object, so the horizon and the target describe the same legs instead of two windows. The per-rung ratio is the floor raised toward median-leg/stop, snapped DOWN to a coarse ladder (2.0/2.5/3.0/4.0/5.0). The ladder is coarse on purpose: PooledGate pools only instruments whose structural break-even matches, and continuous per-instrument ratios would never match and would silently empty the pool. A leg is the right yardstick precisely because it owes NOTHING to the barrier - sizing a target off travel measured over the barrier's own horizon is the circular loop that ran EURUSD/USDCAD away to 14-31*ATR in 2026-08-07. The raise stays bounded by the three tests already in the ladder: reachability, the horizon ceiling (first-passage time grows with stop x target), and the cost fraction. Consequential fixes: the reachability floor was a macro keyed to the fixed ratio and is now BarrierMinReachPct(rr) evaluated per rung (a raised ratio has a lower break-even, so a fixed floor would be the wrong strictness); the detectability break-even likewise; PooledGate now writes and matches the ACTUAL ratio (TargetRR()) rather than the floor. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 16:16:33 -04:00
double m_swingMedianLegAtr;
fix: refuse invalid SL/TP, fix the unreachable deploy floor, scale the horizon Three defects found by reading the 2026-08-01 training logs, all of which only became visible because the relabel made the numbers mean something. 1. A STALE ENUM TRAINED FOUR MODELS ON THE WRONG TARGET. `OnInit: trade settings snapshot - SL_Mode=1 TP_Mode=-101` -101 was TP_PREV_SWING, deleted from TAKE_PROFIT_MODE on 2026-07-31 in 7eb48f5. MetaTrader does not validate a saved enum input against the enum's current members, so charts saved before that kept the old integer. BarrierMultiples()'s `if(tpMult <= 0.0) tpMult = slMult;` then quietly turned it into a 1:1 barrier, and all four topologies trained ~250 eras against a strategy nobody selected - while the log reported "target 1.00*ATR" as though it were configured. Since the relabel these two inputs ARE the label definition, so this is not a bad trade setting, it is a wrong dataset. ValidateBarrier- Inputs() now refuses to start (INIT_FAILED + Alert + an explicit fix) on any value that is not an enum member. Members are enumerated rather than range-checked because both enums are sparse and carry negative sentinels, so no min/max test can tell a legal value from a deleted one - which is the entire failure mode. The fallback survives as belt-and-braces but now announces itself: a fallback that cannot say it fired is indistinguishable from correct behaviour. 2. THE DEPLOYABILITY FLOOR BECAME MATHEMATICALLY UNREACHABLE. `tradeableOK` required `dirPrecPct >= baseRatePct`, where baseRatePct is Buy+Sell as a share of all bars. At the old exact-pivot target that was ~6%, so "beat the base rate" read as "beat chance" and the test looked sound. Triple-barrier labels put it at ~83%, so the gate now demanded 83% directional precision - impossible by construction. Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing safe to deploy" at a perfectly healthy 43-45% precision, with no checkpoint able to ship however good it got. Replaced with ZERO-SKILL precision, max(Buy,Sell)/allBars: exactly the score of the degenerate always-call-one-direction model this floor exists to reject. Correct at any base rate - ~43% on the current labels, ~3% on the old rare-pivot ones. The era line now prints "(chance N%, edge +Mpp)" beside the selection score, because 44% precision is excellent against a 3% chance level and worthless against a 43% one, and reading the first as the second is what made tonight's run look better than it was. 3. THE HORIZON IGNORED THE BARRIER GEOMETRY. ComputeBarrierHorizonBars() returned the median ZigZag leg, which measures how long a ~1 ATR move takes and says nothing about how long the CONFIGURED barrier needs. First-passage time out of [-m,+k] scales with m*k, so a 1:3 barrier takes ~3x as long as 1:1; the unscaled horizon would have timed out most 1:3 trades and pushed Neutral straight back up, re-creating the imbalance the relabel removes. Now multiplied by slMult*tpMult, calibrated against a real measurement rather than assumed: the accidental 1:1 run resolved at horizon 12 with only 16.7% timeouts, so the swing median is the right scale at m*k=1. Verifiable, not just asserted: the prebuild now counts barriers that ended on the VERTICAL barrier and reports them as a share of Neutral. Neutral conflates "timed out" with "stopped out" and only the first indicts the horizon. Both builds compile 0 errors / 0 warnings. Forces a retrain - correcting TP_Mode re-keys the fingerprint (|TB:1:-101 -> |TB:1:3), which is right: no existing model was trained on the intended target. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:30:49 -04:00
int m_labelPrebuildTimeoutCount;
feat(labels): the scheduled close-all is now a vertical barrier in the label walk User report: "I exit everything on Friday close to avoid weekend swap... if the NN training thinks I hold over the weekend it could produce inaccurate results" - it thought exactly that. TripleBarrierLabel walked its full horizon (64 bars, mean lifespan ~18 H4 bars ~ 3 days) straight through the scheduled flat, scoring trades the deployed EA is guaranteed to have closed on Friday 23:45. SQX applies this rule when building strategies; the EA's own labels did not. NextScheduledCloseAll() mirrors CExpertCustom::OnTick's live check exactly (same three inputs, same -1 disabled sentinels, same CLOSE_EVERYDAY semantics, same server clock). The walk stops at the first bar that does not END by the cutoff - OHLC cannot order the tradable fraction of a partial bar, and ties go to the refusal, as everywhere in this file. An unresolved trade at the cutoff times out to Neutral, exactly as live would flatten it. Excursions, the first-passage ladder and the label lifespan truncate with the walk, so the DERIVED geometry is automatically sized to the tradable window - a target the flat rule never lets price reach stops counting as reachable. The prebuild census now splits timeouts: "horizon too short?" vs "ended by the scheduled close-all" - different questions, different fixes. Schedule disabled = no cutoff, exactly like live. Models trained under weekend-blind labels are fitted to a different target; charts with the close-all enabled (the default) should be reset to retrain under the honest labels. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 11:17:14 -04:00
int m_labelPrebuildWeekendCutCount;
//--- Bars where BOTH targets were reached, and the same-bar subset that could not be ordered.
fix: both-won bars were labelled "do not trade" - resolve by first touch Removing the min-reward:risk raise let the MEASURED geometry come back with the target NEARER than the stop (SP500 H1: target 1.62*ATR at q50 of favourable, stop 3.33*ATR at q75 of adverse). That reopened a branch the code called unreachable: price can reach +target and -target inside one horizon, winning in BOTH directions, and those bars fell through to Neutral. Neutral has only three producers, both-lost is unreachable (you cannot touch -3.33 without crossing -1.62 first, which wins the short), and timeouts logged at 1.0% of Neutral - so ~27% of ALL bars were being handed to the model as the abstain class when a trade either way would have collected its target. The cleanest positives in the sample, labelled "do not trade", while the fitted confidence threshold was being asked to find selectivity in what was left. Resolved by FIRST TOUCH: the target reached earlier is the trade that would have closed first. Same forward window, no extra lookahead. Same-bar ties stay Neutral - OHLC cannot order two touches, and unlike an intrabar stop tie there is no pessimistic side to fall to, so a guess would inject a coin-flip direction into the target. Also: - count both-won and its same-bar tie subset in the prebuild line, so the share is measured rather than inferred from arithmetic on a log line - scope the timeout counter to IS, matching the tally it is reported as a percentage OF; it was incremented over the whole scan and divided by an in-sample denominator - clear m_lastBarrierTimedOut at the top of the walk with the excursions, not at the bottom - the two early returns published the previous bar's verdict - mark the pass-1 label line PROVISIONAL. It prints the enum fallback because geometry can only be derived from excursions that do not exist yet, and it reads exactly like a config change that failed to take effect FORCES RETRAIN. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:35:42 -04:00
int m_labelPrebuildBothWonCount;
int m_labelPrebuildBothWonTieCount;
//--- safety valve: Train()'s do-while loop has no other bound on how many eras it will run
//--- before giving up, so a config that can't reach the convergence objective (e.g. too few
//--- swing-confirmed examples for the min recall bar to be reachable) would otherwise loop
//--- forever, permanently keeping the era-progress status label up instead of the normal per-tick
//--- info line and burning CPU nonstop. When the cap is hit, the operator is prompted (see
//--- PromptContinuePastEraCap): CONTINUE resets the era counter and keeps training; STOP deploys
//--- the best checkpoint found so far (FinalizeTrainRun) and terminates training. Headless
//--- (tester/optimizer) runs can't prompt, so they take the STOP branch automatically.
int m_maxErasPerRun;
//--- Train() runs its per-bar loop synchronously, and MQL5 is single-threaded per chart - a
//--- multi-minute era would otherwise starve the terminal's chart-event queue for that whole
//--- stretch, including the control panel's own click/drag hit-testing (Panel\ControlPanel.mqh),
//--- which depends entirely on CHARTEVENT_MOUSE_MOVE being delivered promptly.
bool m_trainRunActive; // true: a run (schedule -> convergence/stop) is in progress, possibly spanning many Train() calls
bool m_eraResumePending; // true: yielded mid-bar-loop last call - resume the SAME era, don't start a new one
//--- One writer for all five, so a yield point cannot save a partial context.
void StashEraResume(const int bars, const int totalIter, const int oosCutoff,
const bool add_loop, const int barIndex);
int m_resumeBars;
int m_resumeTotalIter;
int m_resumeOosCutoff;
int m_resumeBarIndex;
bool m_resumeAddLoop;
//--- Bars pass 1 queued as IS-eligible, trained on in pass 2 in a freshly shuffled order rather
//--- than pass 1's chronological one.
int m_isTrainQueue[];
int m_isTrainQueueCount;
//--- NO PARALLEL WEIGHT/PRIMARY ARRAYS. Removed 2026-08-20; the queue is one bar per slot.
int m_isTrainCursor;
//--- true: pass 1 (sequential) has finished for this era and pass 2 (shuffled backProp) is either
//--- running or has yielded mid-queue - Train() skips straight past pass 1's loop on resume when
//--- this is set. Reset to false only at a fresh era's start (never mid-run).
bool m_isPass2Active;
//--- true: pass 2 has already run to natural completion for this era (m_isPass2Active's own
//--- false state is ambiguous between "not started yet" and "already finished" - both look
//--- identical to a plain `if(!m_isPass2Active)` check).
bool m_isPass2Done;
//--- Pass 3: chronological, OOS-region-only re-walk that happens AFTER pass 2 has actually trained
//--- on this era's IS data - see m_isTrainQueue's declaration comment for why OOS scoring can no
//--- longer just happen inline during pass 1 (that would score every era's OOS window against
//--- weights from BEFORE this era's training, one full era stale - and for era 0 specifically,
//--- against the still-untrained cold-start network, which is why era 0's OOS recall used to show
//--- a meaningless 100% Neutral / 0% Buy / 0% Sell every time). Cursor walks i downward from
//--- m_oosScoreStartIndex to 0, mirroring pass 1's own iteration bounds/order for whichever bars
//--- satisfy isOOS - order matters here (unlike pass 2) since dOosForecast/dOosError are recursive
//--- EMAs over the visitation sequence, not order-independent.
bool m_isPass3Active;
int m_oosScoreIndex;
int m_oosScoreStartIndex;
//--- Pass 2.5: the CALIBRATION walk.
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
bool m_isCalibActive;
bool m_isCalibDone;
int m_calibIndex;
int m_calibStartIndex;
//--- EXCURSION-SIZE HEAD. Kept separate, the classifier is bit-for-bit unaffected and this whole
//--- instrument is removable without trace.
feat: excursion-size head (Stage 1, measurement only) Direction is closed - normalised asymmetry fails on three instruments with a working positive control, and the classifier's own best-of-999 era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is a different question and RANGE clears at ~4x its null. Checked the denomination before building on that, since the source memo warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is predictable" is a claim about travel RELATIVE to current ATR, not a restatement of "ATR is autocorrelated". It is exactly the part a fixed multiple (stop 3.31*ATR, target 1.64*ATR) discards. A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches ladder rung k) upward and downward. Survival parameterisation rather than regressing the multiple, because it needs nothing new from CNet: sigmoid outputs and the per-neuron delta the `total != 3` branch already applies (a quantile head would need a linear activation and a pinball gradient in Network.mqh, Network.cl and the DirectML path, on a class four topologies share). Targets are free - m_ladderUpAt already records first-touch age per rung with 0 meaning never reached. Separate net, not extra outputs on the classifier: more outputs would change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push the count off 3 - the exact condition backProp uses to select the joint softmax gradient the 3-class head depends on. The classifier is bit-for-bit unaffected and this is removable without trace. STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the constant per-rung base rate - the baseline a fixed ATR multiple already assumes - with both predictors fitted IS and evaluated OOS, so neither gets a look at the test set. Positive skill justifies Stage 2 (drive SL/TP and sizing off ExcursionQuantile, which is defined and deliberately uncalled). Zero or negative means ATR already carries everything and Stage 2 must not be built. Trains only on primary occurrences: the replay queue oversamples for CLASS balance, and a direction-balanced sample is a biased SIZE sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
CNet *m_excNet;
bool m_excHeadFailed; // one-shot: creation failed, do not retry every bar
//--- Allocated once, reused every bar. getResults takes CArrayDouble*& and allocates when handed a
//--- NULL, so locals would mean an allocation per bar across ~32k bars an era.
CArrayDouble *m_excTgt;
CArrayDouble *m_excOut;
long m_excBaseHits[2 * BARRIER_LADDER_COUNT];
long m_excBaseTotal; // rows the base rates were estimated from
double m_excBrierHead[2 * BARRIER_LADDER_COUNT];
double m_excBrierBase[2 * BARRIER_LADDER_COUNT];
int m_excScored; // held-out bars scored this era
//--- Since e2c9593 every scored bar IS a disjoint window (the score step strides by the
//--- horizon), so m_excBrierHead/m_excOosHits are already the disjoint tally and m_excScoredD
//--- just counts it.
fix: the trailing incumbent read the future across eras; cold AD blocks cached zeros as truth Three findings from the 2026-08-11 audit: 1. The excursion head's trailing-quantile ring was deliberately never cleared between eras ("a rolling estimate of the market, not of the era") - but pass 3 re-walks the SAME OOS window every era, so at each walk's restart the ring still held the outcome masks of the newest OOS bars from the previous walk: the chronological FUTURE of the bars about to be scored. For the first ~window+horizon pushes of every era the "trailing" incumbent was partly a leading one - conservative for the gate (an informed incumbent is a harder hurdle) but exactly the self-made-artifact class 06d4785 hunts. The ring now clears at era-score reset; the warm-up bars simply don't score the trail race, which the m_excTrailN gating already accounts for. 2. skillTrail compared the head's FULL-block Brier (pro-rated by coverage) against the incumbent's subset sum - valid only if head skill is uniform across the OOS walk, while the trail-scored subset systematically excludes each era's warm-up bars. The audit also found m_excBrierHeadD/BaseD/ m_excOosHitsD declared, zeroed and never accumulated (dead since e2c9593 made every scored bar disjoint). The dead trio is replaced by m_excBrierHeadT: the head's Brier accumulated only on the bars the warm incumbent also scored, so the race now compares both predictors on an identical bar set. 3. The AD/Wyckoff feature blocks read GetData with no EMPTY_VALUE guard; a cold (still-calculating) indicator returns EMPTY_VALUE everywhere, the sanitize loop rewrote that to 0.0, and the bar SUCCEEDED - so BufferTempData cached an all-zero Wyckoff block as a success for the whole bar frame: the one path the f6150ee only-cache-successes rule cannot see, because it never fails (the ba13eef class, arriving through values that never fail; a resumed model's era-0 prebuild starts milliseconds after OnInit). ADIndicatorCold() probes the NEWEST bar - EMPTY_VALUE there means async warm-up (transient reject, retried), while deep bars beyond the buffered depth keep the sanitize loop's neutral-fill so degraded history still trains. Also fixed m_featureCacheValid's declaration comment, which still described the pre-f6150ee cached-miss semantics. Compile: 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 18:20:24 -04:00
double m_excBrierHeadT[2 * BARRIER_LADDER_COUNT];
fix: the excursion gate would have passed Stage 2 on an artifact I made Second-opinion review killed the +4.2% far-rung result, correctly, and the mechanism is my own bug. A head trained toward {0.05,0.9} converges to 0.05+0.85p, so its bias is 0.05-0.15p: negative where p is near 1, POSITIVE where p < 1/3, growing monotonically as the rung gets farther. Against a baseline frozen at the IS rate, an upward-biased head scores positive Brier skill whenever the OOS rate merely sits above the IS rate. Predicted signature: huge negatives near, ~zero at p=1/3, growing positives far. Observed: -82% ... -0.6% ... +1.2/+2.7/+4.2. The far rungs were not the clean end of a distorted measurement, they were the other face of the same artifact. Everything before 25aca83 is void. The gate was a bare `skill >= 2%` point estimate over 8 rungs x 4 topologies x N eras, reported per era - a best-of-~300 with no interval and no multiplicity control, which is the shape of the four traps already documented here. It now needs FOUR things at once: DECISION RUNGS only the rungs ExcursionQuantile actually reads at the live geometry (target 1.62, stop 3.31 ATR), fixed before looking. Skill at 5 ATR is skill about a distance no order is placed at - and the TARGET side currently interpolates 1.5/2.0, which measured -2.2% and -1.3%. DISJOINT SAMPLE one bar per horizon. Adjacent bars share 63 of 64 horizon bars, so ~16k scored bars is ~250 independent ones and every SE over the full set is ~8x understated. VS ORACLE the best constant achievable ON THE SCORED BLOCK, closed form from H and n (Brier = H*(1-H/n)). A head that learned only a LEVEL nearer the OOS rate than the frozen IS constant scores positive against the old baseline and <= 0 here. This is the control that separates per-bar skill from base-rate drift. MONOTONE CURVE P(reach k) must be non-increasing in k. Nothing constrained 8 independent sigmoids to obey that, and ExcursionQuantile returns the FIRST crossing - so a tangled curve is misread exactly where the head is least sure. Counted and reported, not silently used. The pass message now also states what a pass would and would not buy: expectancy is -costs at zero directional edge whatever the stop distance, and under prop DD limits LOWER variance also lowers P(reach target before limit), so "better drawdown" is a choice of failure mode, not a win. Still owed before any Stage 2: a race against a trailing-quantile incumbent and a vol-feature logistic. Beating a frozen global constant is the weakest admissible bar for replacing a global constant. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:49:32 -04:00
int m_excScoredD;
//--- PAIRED PER-WINDOW BRIER DIFFERENCES over the decision rungs - base minus head, and trail
//--- minus head - one value per DISJOINT window.
fix(excursion): the sample gate asked for more windows than the configuration can contain EXCURSION_MIN_DISJOINT was 200, sized as "~16k scored bars over a 64-bar horizon leaves ~250 independent ones". The head scores the OOS SLICE, not the history. At a 30% split and a 32-bar horizon the ceiling is 4691/32 = 147, so 200 was unreachable and every era printed "[disjoint sample too small]", which an operator reads as "wait longer". Unpassable by construction - the identical failure this file already documents one gate down, one layer up. The trail gate inherited it: m_excTrailScored is a subset of the disjoint bars, so it failed the same 200 for the same reason, at 130. Raising OOSSplit or shortening the horizon would clear it and would be fitting the experiment to the answer. Instead, ask the question the count was standing in for - is the skill bigger than its own noise: - The scorer banks one paired Brier difference per DISJOINT window over the decision rungs (base-head, and trail-head). Disjoint by construction, so no EffectiveSampleSize deflation applies - striding by the horizon is what buys that - and paired on identical bars, so the correlation between the two predictors cancels instead of needing to be estimated. - passDj and passTrail now require skill >= EXCURSION_SKILL_USEFUL_PCT AND >= 2 sigma, with the count reduced to a sanity floor of 30. - Both sigmas print on the verdict line. This is not a lowered bar. The 2% skill requirement, the oracle control and the monotone test are untouched, and the sigma test can fail where the count test never spoke: if +8.5% is noise across 147 windows, it will now say so. DecisionRungMask() is the single definition of "rung the decision depends on", called by both the scorer and the report, so the standard error is computed over exactly the rungs the skill score is. The report's inline copy of the bracketing test is gone. Also prints whether the disjoint count is BELOW ITS CEILING or at it, so "not enough yet" and "not in this configuration" stop reading the same. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 23:06:49 -04:00
double m_excDiffSum;
double m_excDiffSumSq;
double m_excTrailDiffSum;
double m_excTrailDiffSumSq;
//--- Which ladder rungs bracket the live SL/TP, i.e. the ones ExcursionQuantile would actually read.
//--- ONE definition, called by both the scorer and the report - they disagreed silently the moment
//--- there were two copies of the bracketing test, and the scorer's copy decides what the report's
//--- standard error is computed over.
void DecisionRungMask(bool &mask[]);
fix: the excursion gate would have passed Stage 2 on an artifact I made Second-opinion review killed the +4.2% far-rung result, correctly, and the mechanism is my own bug. A head trained toward {0.05,0.9} converges to 0.05+0.85p, so its bias is 0.05-0.15p: negative where p is near 1, POSITIVE where p < 1/3, growing monotonically as the rung gets farther. Against a baseline frozen at the IS rate, an upward-biased head scores positive Brier skill whenever the OOS rate merely sits above the IS rate. Predicted signature: huge negatives near, ~zero at p=1/3, growing positives far. Observed: -82% ... -0.6% ... +1.2/+2.7/+4.2. The far rungs were not the clean end of a distorted measurement, they were the other face of the same artifact. Everything before 25aca83 is void. The gate was a bare `skill >= 2%` point estimate over 8 rungs x 4 topologies x N eras, reported per era - a best-of-~300 with no interval and no multiplicity control, which is the shape of the four traps already documented here. It now needs FOUR things at once: DECISION RUNGS only the rungs ExcursionQuantile actually reads at the live geometry (target 1.62, stop 3.31 ATR), fixed before looking. Skill at 5 ATR is skill about a distance no order is placed at - and the TARGET side currently interpolates 1.5/2.0, which measured -2.2% and -1.3%. DISJOINT SAMPLE one bar per horizon. Adjacent bars share 63 of 64 horizon bars, so ~16k scored bars is ~250 independent ones and every SE over the full set is ~8x understated. VS ORACLE the best constant achievable ON THE SCORED BLOCK, closed form from H and n (Brier = H*(1-H/n)). A head that learned only a LEVEL nearer the OOS rate than the frozen IS constant scores positive against the old baseline and <= 0 here. This is the control that separates per-bar skill from base-rate drift. MONOTONE CURVE P(reach k) must be non-increasing in k. Nothing constrained 8 independent sigmoids to obey that, and ExcursionQuantile returns the FIRST crossing - so a tangled curve is misread exactly where the head is least sure. Counted and reported, not silently used. The pass message now also states what a pass would and would not buy: expectancy is -costs at zero directional edge whatever the stop distance, and under prop DD limits LOWER variance also lowers P(reach target before limit), so "better drawdown" is a choice of failure mode, not a win. Still owed before any Stage 2: a race against a trailing-quantile incumbent and a vol-feature logistic. Beating a frozen global constant is the weakest admissible bar for replacing a global constant. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:49:32 -04:00
//--- OOS positives per rung. Feeds the ORACLE control: the best constant achievable ON THE SCORED
//--- BLOCK, in closed form. Separates "predicts per bar" from "learned a level nearer the OOS rate
//--- than the frozen IS constant", which scores positive while carrying no per-bar information.
long m_excOosHits[2 * BARRIER_LADDER_COUNT];
//--- Bars whose predicted survival curve rose with distance. P(reach k) must be non-increasing in k;
//--- nothing constrains 8 independent sigmoids to obey that, and ExcursionQuantile reads the first
//--- crossing, so a tangled curve is misread exactly where the head is least certain.
int m_excMonoViol;
perf: the excursion head cost 3.6x era time - cut its dispatches ~250x Measured on exc-race-v3: LSTM era 300s -> 1087s (net 272->748s, "other" 30->337s). My estimate had been "single-digit percent". The cost is per-DISPATCH, not per-FLOP, and therefore hits EVERY backend: the head is 19k weights and ~2.4 GFLOP an era - seconds of arithmetic - but ~48k forward/backward calls x several layer submits each, and its 760-wide layer exceeds the CPU DLL's inline threshold so each one pays a real handoff. The classifier's own net time tripled too, from contention with a second pool on an already-full box. Three changes, all backend-neutral because they remove submits rather than tune threads: SCORE ONLY DISJOINT WINDOWS (~64x). Adjacent bars share all but one bar of their horizon, so 16k consecutive bars were always ~250 independent observations - the full-sample tally was never worth more than the disjoint one, it just quoted an n that was ~64x too large. Dropping it costs nothing statistically and removes 63 of every 64 forward passes. The two parallel tallies collapse into one, which is also less code. The trailing ring still advances on every bar: it needs the outcome SEQUENCE, and that is array lookups, not a forward pass. TRAIN ON EVERY 4th PRIMARY BAR (4x). The target is low-dimensional and strongly autocorrelated - neighbouring bars carry near-identical excursion information - so per-bar training buys resolution the target does not have. Strided on ATTEMPTS, not acceptances, so a stretch of unlabelled bars cannot silently change the spacing. OWN TIMING COLUMN. The head's passes were landing in the era line's "other" bucket, which is how a 3.6x regression read as an unexplained jump in the one column nobody attributes. A cost that cannot be seen in the timing line cannot be traded off against anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:38:29 -04:00
long m_excTrainTick; // stride counter, on attempts not acceptances
ulong m_excUs; // head's own microseconds this era - see the era line
//--- TRAILING CLIMATOLOGY (see EXCURSION_TRAIL_WINDOW). Ring of per-bar outcome bitmasks - 32
//--- rungs fit one ulong, so the whole rolling history is one array of longs.
feat: race the excursion head against a trailing-quantile incumbent Beating a frozen global constant is the weakest admissible bar for replacing a global constant. The honest incumbent is a rolling rung frequency: it adapts to the volatility regime - exactly what the head claims to predict - and needs no model, no 760 inputs and no training. Implemented as a ring of per-bar outcome bitmasks (32 rungs fit one ulong), sized horizon + EXCURSION_TRAIL_WINDOW. The newest `horizon` entries are held back UNRESOLVED: a bar's rung outcomes are only known one horizon later, so using them would be lookahead and would flatter the incumbent into an opponent the head could never fairly beat. Pass 3 walks oldest-to-newest, so "pushed more than horizon bars ago" is exactly "resolved by now". Each push is O(rungs), not O(window). The head's decision-rung Brier is pro-rated to the trailing estimate's coverage before the ratio, since the incumbent only scores bars where its window is warm. This line is worth reading on its own, independently of the head: if the trailing quantile beats the global constant, that is a cheap risk-control win available with no machine learning at all - and it is the same number either way, so the run answers both questions in one pass. The ring is deliberately NOT reset per era - it estimates the market, not the era, and re-warming 500 bars every era would leave the incumbent unusable over the first chunk of every scoring pass, handing the head a free win on exactly those bars. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:57:11 -04:00
ulong m_excTrailRing[];
int m_excTrailHead; // next write position
int m_excTrailCount; // entries pushed so far, capped at the ring size
long m_excTrailHits[2 * BARRIER_LADDER_COUNT];
long m_excTrailN; // resolved bars currently inside the window
double m_excBrierTrail[2 * BARRIER_LADDER_COUNT];
long m_excTrailScored; // bars scored while the trailing estimate was usable
datetime m_lastBarTime;
//--- This model's own learning-rate trajectory. g_eta is one file-scope global shared by every
//--- CNet in the process, so one member's era-end decay silently changed the rate the OTHER
//--- members' next backProp() used - an unintended coupling between independent trajectories.
double m_modelEta;
//--- Ceiling the era-end recovery bump (Train()'s isBetterEra block) restores `g_eta` toward -
//--- used to be the raw AdamLearningRate unconditionally, which is only correct for ADAM.
double m_etaCeiling;
int m_erasSinceCooldown; // eras completed since the last cooldown reset - replaces the old per-call-only "erasThisCall"
CArrayDouble m_oosWindow; // run-scoped OOS stability window (used to be a Train()-local CArrayDouble)
double m_bestOosForecast;
//--- Balanced accuracy (macro-recall: mean of Buy/Sell/Neutral OOS recall) of the era the
//--- current checkpoint was taken from.
double m_bestBalancedOos;
//--- whether the era m_bestOosForecast/the checkpoint was taken from also cleared the per-class
//--- directional recall floor (see directionalRecallOK below) - part of the "best" ranking itself,
//--- not just a side note, so blended accuracy alone can never outrank a directionally-useful era
//--- (see the checkpoint/g_eta-decay comment in Train()'s era-end block for why that matters).
bool m_bestPassedRecall;
//--- Was the checkpointed era calling BOTH directions? Middle tier of the ranking key - see
//--- isBetterEra.
bool m_bestBothSidesLive;
//--- SLOW-ERA HEARTBEAT (2026-08-10).
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
uint m_eraStartTick;
ulong m_passFeatUs; // cumulative BuildFeatureWindow time this era, microseconds
ulong m_passNetUs; // cumulative feedForward/backProp time this era, microseconds
int m_passHeartbeatPrints;
uint m_lastHeartbeatTick;
//--- How many of pass 1's bars produced a usable feature window, and how many did not.
int m_passWindowOk;
int m_passWindowFail;
fix: the recall gate was unsatisfiable and the LR decay was a spiral Both made the run structurally unable to succeed, independently of any signal in the data. Found by reading the 13:01 log. RECALL GATE. m_objectiveMet required Buy, Sell AND Neutral OOS recall each >= 40%. First-touch resolution (ce52654) collapsed Neutral from the ~94% majority it was under exact-pivot labels to a same-bar-tie residue - 250 of 38,261 bars, 0.65% - so the floor was asking the model to identify 40% of coin-flip ties before it could converge. Measured: CONV, LSTM and HYBRID all logged "Neutral:0% (need >=40% each)" on every era. No model could ever satisfy it; every run was destined for the plateau ladder or the era cap. Only the DIRECTIONAL floors are load-bearing for the anti-collapse job the gate exists to do: an all-Neutral model shows Buy and Sell recall at 0% and is blocked by them. Neutral's own floor guarded the mirror bias (over-calling Buy/Sell at Neutral's expense), which was real at 94% prevalence and is not at 0.65% - there, almost never calling Neutral is correct rather than biased. Prevalence-guarded rather than hardcoded off, so it returns by itself if a future label rule makes Neutral substantial again. Deliberately NOT extended to Buy/Sell: exempting a thin directional class reopens the era-44-46 hole, which directionalRecallMeasured only half-covers - it checks those classes were MEASURED, not that they passed. ETA DECAY. A regressing era restored the checkpoint, reset the optimizer and cut eta - all on the FIRST regression. The next era then started from an identical state with a smaller step, regressed again, and got the same treatment. The loop is self-sustaining and cannot discover anything, because rolling the weights back is exactly what removes the exploration that would end it. Measured on PAI: eras 2-11 every one a regression against era 1, eta 0.000594 -> 0.000024, dW/W 0.000%/0.000% from era 2 onward. Ten eras, ~45s each, reproducing era 1 exactly and unable to do anything else. Now requires ETA_DECAY_PATIENCE_ERAS consecutive regressions - the standard ReduceLROnPlateau formulation. A single bad era is noise, and an improving era clears the counter so alternating runs never accumulate into a decay. Build tag -> gate-patience-v3. It had not moved in six commits, which is why the running binary could not be identified from its own log. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:28:58 -04:00
//--- Consecutive regressing eras since the last new best - the patience counter for the checkpoint
//--- restore / g_eta decay (see ETA_DECAY_PATIENCE_ERAS). Reset by any era that improves.
fix: the recall gate was unsatisfiable and the LR decay was a spiral Both made the run structurally unable to succeed, independently of any signal in the data. Found by reading the 13:01 log. RECALL GATE. m_objectiveMet required Buy, Sell AND Neutral OOS recall each >= 40%. First-touch resolution (ce52654) collapsed Neutral from the ~94% majority it was under exact-pivot labels to a same-bar-tie residue - 250 of 38,261 bars, 0.65% - so the floor was asking the model to identify 40% of coin-flip ties before it could converge. Measured: CONV, LSTM and HYBRID all logged "Neutral:0% (need >=40% each)" on every era. No model could ever satisfy it; every run was destined for the plateau ladder or the era cap. Only the DIRECTIONAL floors are load-bearing for the anti-collapse job the gate exists to do: an all-Neutral model shows Buy and Sell recall at 0% and is blocked by them. Neutral's own floor guarded the mirror bias (over-calling Buy/Sell at Neutral's expense), which was real at 94% prevalence and is not at 0.65% - there, almost never calling Neutral is correct rather than biased. Prevalence-guarded rather than hardcoded off, so it returns by itself if a future label rule makes Neutral substantial again. Deliberately NOT extended to Buy/Sell: exempting a thin directional class reopens the era-44-46 hole, which directionalRecallMeasured only half-covers - it checks those classes were MEASURED, not that they passed. ETA DECAY. A regressing era restored the checkpoint, reset the optimizer and cut eta - all on the FIRST regression. The next era then started from an identical state with a smaller step, regressed again, and got the same treatment. The loop is self-sustaining and cannot discover anything, because rolling the weights back is exactly what removes the exploration that would end it. Measured on PAI: eras 2-11 every one a regression against era 1, eta 0.000594 -> 0.000024, dW/W 0.000%/0.000% from era 2 onward. Ten eras, ~45s each, reproducing era 1 exactly and unable to do anything else. Now requires ETA_DECAY_PATIENCE_ERAS consecutive regressions - the standard ReduceLROnPlateau formulation. A single bad era is noise, and an improving era clears the counter so alternating runs never accumulate into a decay. Build tag -> gate-patience-v3. It had not moved in six commits, which is why the running binary could not be identified from its own log. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:28:58 -04:00
int m_consecutiveRegressions;
void TrainHeartbeat(const string tag, int done, int total, const string shortLabel);
//--- Progress of the pass currently running, and its name, for the simple panel.
int m_passProgressPct;
string m_passLabel;
fix: prebuild and era sized different windows; diag: Train() names its branch TWO things, one incident. 1) THE BUG I SHIPPED IN 0c85c54. m_tuneStartTrainBar is declared, initialised to 0, and NEVER ASSIGNED - the assignment existed before the God-class split and the split dropped it, leaving a dead member. Harmless while nothing read it; a real defect the moment 0c85c54 made StartLabelCachePrebuild() reset dtStudied from it. Train() then computed the window as max(StartTrainBar, floor) while the prebuild computed max(0, floor), where StartTrainBar is the non-zero datetime OnChartEventHandler passes through from the "New Bar" event. The two therefore disagreed about `bars`, so EnsureBarCachesCapacity() saw a changed size at era start, wiped the caches, and re-armed a full 38k-bar prebuild - instead of training. Restored the assignment so both sides evaluate the identical expression. 2) THE REASON IT TOOK ALL NIGHT TO FIND. Train() is a state machine with six early-return branches above the era loop and every one of them is silent. Four charts burned a core each for 15 minutes with an empty journal: the pass heartbeats (694b756) proved the era loop was never reached, no prebuild completion line appeared either, and nothing external can see inside a single MQL5 thread - per-thread CPU says "busy", file writes say nothing, and the VPS has no debugger. That is an undiagnosable state, and it is the thing to fix, not just the bug of the day. ReportTrainStall() now names the branch Train() is taking whenever no era has completed for 3 minutes, at most once a minute per signal, with the state that decides the branch: run/prebuild/simOos/resume flags, era, dtStudied, and - for the cache-invalidation branch specifically - BOTH bar counts, since two sizings disagreeing is exactly what re-arms the prebuild forever. Silent on a healthy run: an era completing resets the clock. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:32:08 -04:00
//--- STALL REPORTER. Train() is a state machine with several early-return branches ABOVE the era
//--- loop (OOS simulation walk, label prebuild, history sync, warm-up, cache invalidation), and
//--- every one of them is silent.
fix: prebuild and era sized different windows; diag: Train() names its branch TWO things, one incident. 1) THE BUG I SHIPPED IN 0c85c54. m_tuneStartTrainBar is declared, initialised to 0, and NEVER ASSIGNED - the assignment existed before the God-class split and the split dropped it, leaving a dead member. Harmless while nothing read it; a real defect the moment 0c85c54 made StartLabelCachePrebuild() reset dtStudied from it. Train() then computed the window as max(StartTrainBar, floor) while the prebuild computed max(0, floor), where StartTrainBar is the non-zero datetime OnChartEventHandler passes through from the "New Bar" event. The two therefore disagreed about `bars`, so EnsureBarCachesCapacity() saw a changed size at era start, wiped the caches, and re-armed a full 38k-bar prebuild - instead of training. Restored the assignment so both sides evaluate the identical expression. 2) THE REASON IT TOOK ALL NIGHT TO FIND. Train() is a state machine with six early-return branches above the era loop and every one of them is silent. Four charts burned a core each for 15 minutes with an empty journal: the pass heartbeats (694b756) proved the era loop was never reached, no prebuild completion line appeared either, and nothing external can see inside a single MQL5 thread - per-thread CPU says "busy", file writes say nothing, and the VPS has no debugger. That is an undiagnosable state, and it is the thing to fix, not just the bug of the day. ReportTrainStall() now names the branch Train() is taking whenever no era has completed for 3 minutes, at most once a minute per signal, with the state that decides the branch: run/prebuild/simOos/resume flags, era, dtStudied, and - for the cache-invalidation branch specifically - BOTH bar counts, since two sizings disagreeing is exactly what re-arms the prebuild forever. Silent on a healthy run: an era completing resets the clock. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:32:08 -04:00
uint m_lastEraCompleteTick;
uint m_lastStallReportTick;
void ReportTrainStall(const string branch);
bool m_haveOosCheckpoint;
bool m_oosStable;
bool m_objectiveMet;
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
//--- RAW inputs to the family-wise deployment gate, snapshotted at the same instant as the checkpoint
//--- so the test re-runs on the era that will actually ship rather than on whatever the latest era
//--- happened to score. m_bestBalancedOos alone cannot serve: it is precision already multiplied by
//--- the coverage credit, and the significance test needs the unweighted precision, the chance rate it
//--- is measured against, and the call count that sets its standard error. -1 until the first ranked era.
double m_bestDirPrecPct;
double m_bestChancePrecPct;
int m_bestDirCalls;
//--- DECLUSTERED OOS tally: the calls that survive NMS, i.e. the ones that actually become
//--- positions now that live NMS gates the trade (see RefreshLatestSignal). Era-scoped, reset
//--- with the rest of the OOS counters.
fix: NMS gates the TRADE, not just the arrow - one arrow is now one trade NmsLiveAccept() appeared in exactly one place: wrapped around DrawObject(). It never touched dPrevSignal, and dPrevSignal is what LongCondition() / ShortCondition() / SignedAIConfidence() read. So a declustered bar lost its arrow and still opened a position. Measured on SP500 H1 2026-08-09: CONV called a direction on 64% of bars, so the ~500 bars visible on screen held ~320 decisions - and ~40 arrows were drawn. Roughly one arrow per eight positions the EA would take. And the survivors are not a random eighth. Rule 2 of the declustering keeps the HIGHER-CONFIDENCE side of a cluster, so the visible set is systematically the best member of each run. A chart showing the best of every eight decisions and hiding the rest reads far better than the model is - the same best-of-N selection error already corrected in the geometry scan, the indicator tuner, the lag profile and the deploy gate, this time on the display layer, where it is most likely to mislead the person deciding whether to trade. Fixed by neutralising dPrevSignal when NMS rejects, rather than adding a "may trade" flag consulted at each read site: that leaves exactly ONE definition of what the model decided this bar, so the arrow, the panel's "Current signal", the confidence feeding sizing/SL/TP/trailing, the refresh tally and the order itself cannot drift apart again. Also reports the consequence instead of hiding it. Every OOS counter on the era line still scores every directional call - a population ~8x larger than what now trades - so the line carries a second figure: | TRADED (declustered) NN% on N calls (edge +Npp) replaying the identical rule over pass 3 (which walks OOS bars oldest to newest, the same order the live sweep sees). Its cursors are separate members from the live ones so a training pass can never disturb the live chart's declustering. Deliberately NOT switched into selectionScore yet. Declustering cuts coverage from ~64% of bars to ~8%, well under MIN_COVERAGE_FRACTION_OF_BASE_RATE, which would make every checkpoint undeployable overnight - the minRR collision and the recall-floor catch-22 twice over. The floor gets re-derived from these measurements first. Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-09 10:22:31 -04:00
int m_oosNmsFired;
int m_oosNmsHits;
int m_oosNmsLastBuyIdx;
int m_oosNmsLastSellIdx;
int m_oosNmsKeptIdx;
double m_oosNmsKeptConf;
ENUM_SIGNAL m_oosNmsKeptDir;
//--- How many eras the maximum was taken over - the N in the Sidak correction. Run-scoped: reset
//--- with the rest of the best-checkpoint tracking at the top of a fresh run.
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
int m_deployCandidateEras;
//--- THE GATE. Re-tests the checkpoint that is about to deploy against the null of the MAXIMUM over
//--- m_deployCandidateEras eras, and reports the pieces so the log can show its working. See
//--- DEPLOY_FAMILY_WISE_ALPHA. Returns false (refuse) whenever the inputs are missing.
bool BestCheckpointSurvivesSelection(double &zObs, double &pFamily, int &nTried);
//--- Logs that verdict WITHOUT enforcing it, for the two deploy paths that are explicit operator
//--- decisions (the era cap and the panel's Deploy button). Those stay the operator's call; this just
//--- makes sure the log never lets an authorised deploy read as a validated one.
void ReportSelectionGateVerdict(string context);
//--- Plateau ladder state (see the PLATEAU_* constants). m_erasSinceBestBalanced counts eras
//--- since the last NEW BEST balanced accuracy; m_plateauStage is how far up the escalation it
//--- has climbed.
int m_erasSinceBestBalanced;
int m_plateauStage;
feat(search): stop on the IN-SAMPLE plateau, and shrink every best-of-K effect before quoting it Points 3 and 4 of the four-point plan. 1. IN-SAMPLE EARLY STOP - and the reason it is worth having is not compute. The plateau ladder stops on the OOS SELECTION score. That is a peek: by the time it fires, every one of those eras has been evaluated out of sample, so all of them sit in the family the deploy gate corrects over (g_ensCandidateEras, Sidak). Training longer therefore does not merely cost time - it RAISES the bar the eventual winner has to clear. The new stop reads the TRAINING error, which the gate never looks at. When the optimiser has stopped improving on data it can see, more eras will not find a better model; they will only enlarge the OOS family. Ending there shrinks the correction, and the shrinkage is legitimate precisely BECAUSE the stopping rule never consulted an out-of-sample number. That distinction is the whole point and it is the one this project has got wrong four times: stop on IS and the family really is smaller; stop on OOS and those eras were searched and still count. Both stops now exist; only this one buys a lower bar. Deliberately more patient than the OOS ladder (IS_ERROR_PATIENCE_MULT = 3x): training error is noisy per era - mini-batch order alone moves it - and ending a run that is still learning costs far more than a few wasted eras. Improvement is RELATIVE (IS_ERROR_IMPROVE_FRAC = 1%), so it does not depend on the loss's absolute scale, and it only acts when a checkpoint exists, since otherwise it would end a run with nothing to deploy. Reset per RUN alongside the ladder, so a resumed run cannot early-stop on its first era against a previous run's best. 2. WINNER'S-CURSE SHRINKAGE ON THE BARRIER-GEOMETRY WINNER. The family-wise permutation gate already establishes that the RANKING is not noise. It says nothing about the SIZE of the winner's effect - and a best-of-K maximum is biased upward by construction, being the largest of K noisy draws. The adoption message quotes that raw maximum and compares it against the incumbent, so the number a reader plans on is the inflated one. The penalty is now measured, not assumed: the same permutation draws that produce the p-value also produce, per draw, the MAXIMUM excess across all candidates under pure noise. The mean of those maxima is exactly what a best-of-K selection is expected to report when there is nothing there. This is the empirical form of the sqrt(2 ln K) x SE penalty the SQX EdgeFinder plugin applies to every maximum it reports (Stats.java:79-88), and it needs no normality assumption because the draws ARE the null distribution. Applied in James-Stein form - effect x max(0, 1 - penalty^2/effect^2) - so a large effect is nearly untouched and a marginal one collapses toward zero. Reported, not gated. The adoption decision still turns on the permutation p-value, which is the right test for "is the ranking real"; the shrunk number is there so the magnitude quoted beside it is one worth planning on. Closes the first of the two EdgeFinder ports identified on 2026-08-12. NOTE on the second EdgeFinder port, deliberately not done here: "let the measurement steer the target" is already true where it matters most - ReportGeometryExpectancyScan ADOPTS the winning barrier geometry under the family-wise gate rather than advising it, and the MI excursion suite publishes a verdict per instrument per config. What is still missing is steering the TRAINING TARGET itself (direction vs excursion) off those verdicts, and that is a design change rather than a surgical one - direction is a closed verdict while excursion SIZE keeps clearing, so the honest version of that change is a target-selection policy, not a flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 16:29:00 -04:00
//--- IN-SAMPLE early-stop state (see IS_ERROR_IMPROVE_FRAC). Best training error seen this run and
//--- eras since it last improved. -1 = nothing measured yet.
double m_bestIsError;
int m_erasSinceBestIsError;
//--- LATCHES when the IN-SAMPLE error stops improving, and it is a separate flag from
//--- m_plateauStage for one measured reason: EnsembleEraVerdict mirrors the shared ladder onto
//--- every member with `mm.m_plateauStage = g_ensPlateauStage` on EVERY era, purely so each
//--- member's status line reads the collective stage.
fix(plateau): the IS-error early stop was inert for every ensemble member 15 hours of training, and the stop that exists to END a run announced itself 1,299 consecutive times without ending anything: SP500 ConvLSTM IN-SAMPLE ERROR PLATEAU - not improved in 1297 / 1298 / 1299 eras (best 0.2689, now 0.3269) ... era 1396, 1397, 1398 SP500 LSTM 536 eras SP500 CONV 442 eras SP500 PAI 150 eras XAUUSD HYB 478 eras XAUUSD LSTM 296 eras XAUUSD CONV 366 eras CAUSE: it wrote its decision into m_plateauStage, and EnsembleEraVerdict mirrors the shared ladder onto every member - `mm.m_plateauStage = g_ensPlateauStage` - on EVERY era, purely so each member's status line shows the collective stage. A display mirror was silently overwriting a decision, so the stop re-armed and re-fired the next era, forever. This is the worst possible direction for this particular bug. Every one of those 1,299 eras was scored out of sample and joined the family the deploy gate corrects over (Sidak, g_ensCandidateEras). The stop's entire purpose is to make that family SMALLER; instead the run spent fifteen hours raising its own bar. - m_isErrorPlateaued: a one-way per-member latch, cleared only by a fresh run. Nothing in the ladder may reset it. The stop condition and the two solo deploy conditions read the latch, not the mirrored stage. - The orchestrator combines: EnsembleEraVerdict requires UNANIMITY across participating members (same participation test the era barrier uses, so an excluded or finished member cannot veto). One member still learning can still move the combined vote, and the vote is what the gate certifies. - Fed in as `dueStage = PLATEAU_STAGE_DEPLOY`, NOT written to g_ensPlateauStage. The block that actually ends the run sits under `dueStage > g_ensPlateauStage`, so assigning the stage directly makes that test false and the deploy never happens - the same inert-write shape as the bug being fixed. Caught before committing; raising dueStage carries it through the ladder's own path (warm restarts skipped, family-wise vote test, measurement screen, joint checkpoint) unchanged. - g_ensIsPlateauAnnounced: announce once per run, not once per era. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 13:09:26 -04:00
bool m_isErrorPlateaued;
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//--- Eras remaining in the current warm-restart boost window (see PLATEAU_RESTART_BOOST): set to
//--- PLATEAU_PATIENCE_ERAS by each boosted restart, decremented by the era-end anneal that walks
//--- g_eta back to the ceiling, cleared by any new best.
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
int m_restartBoostErasLeft;
refactor(ai): nine class-imbalance inputs down to two The imbalance section offered nine controls for one job. Audited against the code, five of them did not do what their names said at the shipped defaults: AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns whenever the adjusted loss is on, which is default. OversampleParity DEAD in training - Training.mqh gated the replay loop on !useLogitAdjustedLoss (correctly, citing Buda et al. 2018). Live only in the online-learning path. EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma damper - "replay minority bars through pass-2 oversampling" was a focal-loss switch. ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25. UseStaticPrior An exact duplicate of FreezePriorCalibration - the two were OR'd together in the single place either is read. So they were not five mechanisms fighting; they were one mechanism plus eight knobs that mostly described machinery that no longer ran. That is worse than a real conflict, because the log agreed with the names: the label-cache line printed "reps up to 28x (90% parity) (seeding era 0's class-balance oversampling)" on every run, describing an oversampling pass that had been switched off. It is fixed here too - it cost this session a wrong diagnosis. The one genuine redundancy was focal loss, running at gamma*0.125 alongside the adjusted loss: two corrections on the same axis, the exact stacking failure this file already cited Buda et al. for in two other places, damped by a replay flag whose replay path was itself dead. Removed rather than re-tuned. The plateau ladder is unaffected - its escape is the learning-rate warm restart; the gamma anneal beside it only ever stepped toward zero. WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze: LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted- Loss boolean, since a strength dial where 0 already means off does not need an on/off switch beside it. FreezePriorCalibration unchanged. It is the only one of the six corrections with a consistency guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection already ranks on - so the loss and the deploy decision optimize one thing. The online continual-learning path keeps its own alpha-balanced focal weight, now as constants pinned to the removed inputs' shipped defaults, so its behaviour is unchanged. It legitimately needs its own correction: ApplyLogitAdjustment() only runs inside a training run, so a deployed model that was reloaded carries no logit offsets and would otherwise stream 31:1 data into itself uncorrected. The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a double fed to a %d conversion and had always emitted a literal 0; the |MR: segment is written as the constant its shipped defaults produced. Dropping either would have re-keyed every model and forced a from-scratch retrain of the one topology currently converged and trading. Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable "neutralized by prior correction" diagnostic. Both builds compile 0 errors, 0 warnings. No retrain forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- m_focalGammaRuntime removed 2026-07-31 with focal loss itself - see the removal note at the
//--- former m_focalGamma above. The plateau ladder keeps its learning-rate warm restart, which was
//--- always the actual escape; the gamma anneal beside it stepped monotonically to zero anyway.
uint m_syncWaitStartTick; // 0 = not waiting on history sync; else GetTickCount() when the wait began
//--- 3 no-op passes on a fresh start (see InitNeuralNetwork()/ResetWeights()), each its own separately-
//--- scheduled Train() call (not a tight in-process loop), so the broker/terminal's history sync gets
//--- several real, wall-clock-separated chances to finish before the era loop commits to a bar count.
int m_warmupPassesRemaining;
//--- fractal/swing-confirmation/trend-context Buy/Sell label cache: the label at a given now-
//--- relative bar index only depends on price/ATR history, never on model state, so recomputing
//--- it every era (as opposed to once per real bar close) is pure waste.
diag: is "optimal SL/TP" learnable? Score the features against excursions Proposed direction: train the net to predict entry/SL/TP that maximise return and minimise drawdown, rather than to classify direction. Before rebuilding a head, measure whether the target is learnable at all. That question splits into two that behave nothing alike: HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility clustering is about the most robust regularity in markets. WHICH WAY it goes first (the asymmetry) - direction, which is what every noise-floor verdict in this project has been about. Expectancy comes ONLY from the second. The first buys position sizing and drawdown control - worth having under prop-firm limits, but not an edge: exit management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with expectancy FLAT. Crucially this is NOT already answered. Every MI figure here scored the triple-barrier label, i.e. one specific question at one fixed geometry. A noise-floor result there says nothing about whether excursion MAGNITUDE is learnable - different target, different answer. Four targets, and the verdict is the CONTRAST, printed explicitly because the dangerous misreading of "UP clears" is "we can predict profitable trades": RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that SHOULD clear. Every prior verdict here lacked a control expected to pass; a range target at the floor indicts the measurement, not the market. UP / DOWN - MFE / MAE. ASYMMETRY - up-dn, the only one that can pay. Collected inside the walk the label already does (one max, one min per bar). The early-out when both barriers resolved is GONE: it would have truncated the excursions at whichever bar tripped the last barrier, making the measurement a function of the CURRENT SL/TP - the circularity this is trying to escape. The loop was already bounded by the horizon, so only the average cost moves. Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is fat-tailed and fixed-width bins would put nearly every row in bin 0; it also pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and to the barrier label's ~1.02 instead of confounded by class balance. Two bugs fixed in this code before it ever ran, both of which would have produced a plausible quiet wrong answer rather than an error: - TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the accumulators were reset, so one bar's excursions would be cached under another bar's index. Cleared at the top now, ahead of every return. - An unresolvable bar is still flagged as labelled but carries excursions of exactly 0. Under equal-frequency binning a block of identical zeros drags the lowest cut onto zero and a third of the sample lands in one uninformative bin - a depressed score that reads as "not predictable", a false negative in the direction that would wrongly kill the idea. Rows where both excursions are zero are dropped; price cannot travel zero both ways over a whole horizon. Read-only diagnostic. No topology or label change: no retrain of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 10:22:41 -04:00
double m_excUpCache[];
double m_excDownCache[];
fix(geometry): a free zero made "never resolve" the winning geometry The CANDIDATE GEOMETRY line shipped in 05f1a53 said per-candidate geometry beats the global pair on every SP500 member at 2-3 sigma. It does not. It said so because a bar that reached neither barrier scored 0 R, and the incumbent's mean is NEGATIVE (-0.07 to -0.21 R). Against a losing baseline a free zero is a win, so the widest candidate always came out ahead - and the reported gain ordered itself by timeout share, not by skill: PAI 95.1% timed out -> +0.189 R (head measured -2.42 sigma, HARMFUL) HYB 73.8% -> +0.182 R (head at chance, +0.68 sigma) CONV 61.8% -> +0.163 R (head measured -2.47 sigma, HARMFUL) LSTM 27.1% -> +0.158 R (head +1.67 sigma) Monotone in the timeout share and inverted against the sigma gate. The acceptance test written when this was built - "the sigma gate predicts LSTM helps and CONV hurts; if the R difference does not reproduce that ordering, something is wrong" - is what caught it. A trade that reaches neither barrier is not worth zero. It is closed at the horizon, which is what the scheduled close-all does live and what SimulateTradeOutcome's timeout path already charges. So mark it there: TripleBarrierLabel now publishes the signed close-to-close travel at the last bar it actually visited (m_termTravelCache, same validity flag as the excursion and ladder caches), and LadderOutcomeR prices a timeout off it instead of returning false. A bar that cannot be evaluated under BOTH pairs is now dropped whole - scoring one leg and defaulting the other is the same bug in a smaller costume. Second defect, same function: CandidateGeometryFor applied neither of the floors the global derivation applies, so on USDJPY it chose stop 2.00 / target 1.00 - a 67% break-even, forbidden by the 1:2 policy floor. c3daded in miniature: a selector optimising its own criterion with no reference to the decision criterion. Both floors now apply, and the ratio is re-checked AFTER the per-leg rung snap, which can lose it. Also: the module weight was an unshrunk pooled win rate. USDJPY ConvLSTM fired 19 times (2.0 effective), won 36.8%, and took module weight 0.37 - 41% of the ensemble's capable weight and the loudest voice on the chart, off two effective observations. It also lifted the computed vote ceiling to 26.3 against a 25 threshold, which is why THRESHOLD UNREACHABLE never printed on a chart whose peak vote is 14 and whose practical ceiling without that member is 18.8. The pooled rate is now shrunk toward the coin-flip rate on the era's own OOS bars over 30 prior-equivalent calls, and the tiers shrink toward the shrunk value rather than the raw one. A member with ~300 effective calls moves by ~0.4pp; the 19-fire member goes 0.37 -> ~0.15. MEASUREMENT ONLY still - no order reads any of this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 11:30:54 -04:00
//--- Published under the same validity flag as the two above - see m_lastTermTravel.
double m_termTravelCache[];
//--- First-passage ladder, flat (idx * BARRIER_LADDER_COUNT + level). Value = bars AFTER the
//--- entry bar at which travel first reached BARRIER_LADDER[level] in that direction; 0 = never
//--- within the horizon.
feat: first-passage ladder + expectancy scan - price every geometry, not just the chosen one Corrects the premise of the previous plan. Break-even is NOT a ceiling. If the model shifts the win probability on the bars it selects from p0 = m/(m+k) to p0 + d, then EV = (p0+d)*k - (1-p0-d)*m = d*(k+m) because p0*k - (1-p0)*m is zero by construction. The stop:target RATIO is expectancy-neutral - a punishing break-even is exactly repaid by the payoff - and only the real edge d and the TOTAL WIDTH (k+m) move EV. Width matters because the spread is charged once per trade however wide the barriers are, so a narrow barrier spends much of its own range on costs. DeriveBarrierGeometry's own comment already said the ratio buys nothing; the objective just never followed from it. Blocker this had to solve first: m_excUpCache/m_excDownCache hold only MAXIMUM travel each way, and a maximum cannot say which side was reached FIRST - so any geometry other than the walked one was undecidable on precisely the bars where both barriers were touched, ~28% of the sample. - BARRIER_LADDER: per bar, the first-touch AGE for 8 travel distances in each direction, filled during the walk the labels already run. Cursors keep it O(1) amortised per walked bar rather than 16 comparisons. Levels are travel FROM ENTRY, not barrier prices, so one ladder serves both directions and the spread is applied analytically when a level converts back to an SL/TP multiple - storing prices would need four ladders and bake today's spread into the cache. Sized, invalidated and validity-gated with the label caches. - ReportGeometryExpectancyScan: every ladder pair priced exactly off that cache - width in ATR and in SPREADS (cost efficiency, knowable without knowing d), break-even, both base rates, the share of bars resolved inside the horizon, and EV per unit of edge. Compares the widest resolvable pair against the quantile rule's pick. MEASUREMENT ONLY - the quantile rule still chooses. Nothing here can measure d, and width buys nothing if the wider target is less predictable. Base rates are printed beside each break-even because a persistent gap is DRIFT and must not be credited to the model. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 12:59:18 -04:00
int m_ladderUpAt[];
int m_ladderDownAt[];
//--- Scratch for the bar TripleBarrierLabel is currently walking, published the same way m_lastExcUp
//--- is and copied into the caches by AdvanceBarrierLabelState under the label's validity flag.
int m_lastLadderUpAt[BARRIER_LADDER_COUNT];
int m_lastLadderDownAt[BARRIER_LADDER_COUNT];
//--- Reports expectancy for every ladder pair - see the definition. Measurement only; it does not
//--- (yet) choose the geometry.
void ReportGeometryExpectancyScan(void);
bool m_labelCacheBuy[];
bool m_labelCacheSell[];
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
//--- Per-bar outcome of each DIRECTION taken on its own, cached beside the label under the same
//--- m_labelCacheHasValue flag. This is what the deploy gate scores against - see
//--- m_oosWinLongTotal.
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
bool m_winLongCache[];
bool m_winShortCache[];
bool m_labelCacheHasValue[];
int m_labelCacheBars; // 0 = no cache built yet
datetime m_labelCacheAnchorTime; // m_Time.GetData(0) at last (re)build - 2nd invalidation key
void ComputeLabelForBar(int i, int bars, bool &buy, bool &sell);
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
void AdvanceBarrierLabelState(int i, int bars);
//--- The triple-barrier verdict for one bar - the training TARGET.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
ENUM_SIGNAL TripleBarrierLabel(int idx);
feat(labels): the scheduled close-all is now a vertical barrier in the label walk User report: "I exit everything on Friday close to avoid weekend swap... if the NN training thinks I hold over the weekend it could produce inaccurate results" - it thought exactly that. TripleBarrierLabel walked its full horizon (64 bars, mean lifespan ~18 H4 bars ~ 3 days) straight through the scheduled flat, scoring trades the deployed EA is guaranteed to have closed on Friday 23:45. SQX applies this rule when building strategies; the EA's own labels did not. NextScheduledCloseAll() mirrors CExpertCustom::OnTick's live check exactly (same three inputs, same -1 disabled sentinels, same CLOSE_EVERYDAY semantics, same server clock). The walk stops at the first bar that does not END by the cutoff - OHLC cannot order the tradable fraction of a partial bar, and ties go to the refusal, as everywhere in this file. An unresolved trade at the cutoff times out to Neutral, exactly as live would flatten it. Excursions, the first-passage ladder and the label lifespan truncate with the walk, so the DERIVED geometry is automatically sized to the tradable window - a target the flat rule never lets price reach stops counting as reachable. The prebuild census now splits timeouts: "horizon too short?" vs "ended by the scheduled close-all" - different questions, different fixes. Schedule disabled = no cutoff, exactly like live. Models trained under weekend-blind labels are fitted to a different target; charts with the close-all enabled (the default) should be reset to retrain under the honest labels. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 11:17:14 -04:00
//--- First scheduled close-all strictly after `after`, 0 when the schedule is disabled - the
//--- label walk's second vertical barrier. Body beside TripleBarrierLabel in AIBase\Labels.mqh.
datetime NextScheduledCloseAll(const datetime after);
diag(barriers): the horizon the geometry is sized for does not exist Every label timeout on both live charts was the scheduled close-all and none was the horizon. Not "mostly" - all of them: USDJPY 14417 of 14417 timeouts ended by the close-all SP500 2434 of 2434 targetDayOfWeek is CLOSE_FRIDAY, so every position is flattened weekly. A trading week is ~30 H4 bars and an entry lands uniformly inside it, so the average bar is labelled under ~15 bars of runway. The horizon ladder granted USDJPY 96 and SP500 32, and the SCALE ladder rejects rungs against BARRIER_HORIZON_MAX (384) - a ceiling that never binds while the one that does is invisible to it. USDJPY's chosen target is 6.00*ATR, asked of a trade that lives ~11 bars: 78.6% of labels come back Neutral, the base rate collapses to 14.0%, and no model can clear a 33.4% break-even against a label that mostly cannot resolve. The close-all itself is correct and must stay - it is what the account actually does, and 3e467f9 put it into the labels for that reason. What is wrong is that the geometry deriver has never been told about it. This commit only MEASURES it. MeasureCloseAllBudget() walks the real bar series (session- and DST-correct, not arithmetic on a nominal week) and returns the cycle length plus the mean an entry gets; a CLOSE-ALL BUDGET line prints both next to what the ladder granted. No geometry changes: the horizon is a label parameter, so capping it re-keys every fingerprint and costs a full retrain on both charts. That is the operator's call, and it should be made against this line rather than against my arithmetic. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 16:24:14 -04:00
//--- MEASURED bars between consecutive scheduled close-alls, and the mean an entry actually gets.
//--- The horizon ladder is capped by BARRIER_HORIZON_MAX, which the close-all makes fiction: no
//--- trade survives one cycle, whatever the ladder granted. Diagnostic only for now.
int MeasureCloseAllBudget(int &meanBudgetBars);
fix(barriers): cap the horizon at what the close-all actually grants The diagnostic shipped in de382bb came back off both live charts and confirmed the arithmetic exactly: CLOSE-ALL BUDGET - flattens every position every 29 bars ... an entry landing anywhere in the cycle gets 15 bars on average. The horizon ladder just granted 128. So the ceiling the ladder was rejecting rungs against - BARRIER_HORIZON_MAX, 384 - never bound anything, while the one that does bind was invisible to it. SnapHorizonToLadder and the scale ladder's fitsH test now both read EffectiveHorizonMax(), which is the measured close-all cycle. One function, so the ceiling cannot be lowered in the snap and left high in the rejection test. The CYCLE, not the 15-bar mean: a Monday entry really does get the whole cycle, and rejecting on the mean would invent a second criterion where the design deliberately has one ceiling and reports the milder snap-down truncation instead of rejecting on it. Expect the ladder to pick a NARROWER pair, which is what the MEASURE objective already asks for - min provable EV grows as width squared, and USDJPY's 6.00*ATR target was being asked of a trade that lives ~11 bars. "Schedule off" is cached; "not enough bars loaded yet" is not. Caching the latter would restore the 384-bar ceiling for the whole process because one early call landed before history arrived. RE-KEYS EVERY FINGERPRINT - the horizon is a label parameter, so this is a full retrain on both charts. Done now because both are at era 0 after a fresh deploy, which is the cheapest this change will ever be. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-23 08:32:13 -04:00
//--- BARRIER_HORIZON_MAX, lowered to what the scheduled close-all actually grants. THE horizon
//--- ceiling from 2026-08-22 on: every label timeout on both live charts was the close-all and
//--- none was the horizon, so the old ceiling never bound anything. Measured once, then cached -
//--- the scale ladder asks per rung.
int EffectiveHorizonMax(void);
int m_closeAllCycleBars; // 0 = not measured yet, -1 = no schedule
int m_closeAllMeanBudget;
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- Resolves the SL/TP ATR multiples the label uses from the EA's live SL_Mode/TP_Mode. Split out
//--- because the INTELLIGENT modes scale with AI confidence, which does not exist at label time -
//--- see the definition for why the label uses their zero-confidence base instead.
void BarrierMultiples(double &slMult, double &tpMult);
//--- Long win-share at (sl, tp) from the first-passage ladder - the FULL horizon, not the
//--- excursion cache's much shorter reference window. Callers that compare rungs to each other
//--- must log them - see the definition.
fix(labels): overlapping-label sample correction + horizon cap on the scale ladder Three defects, all surfaced by the 2026-08-17 SP500 H4 run that shipped stop 4.86 / target 9.71 (width 14.57*ATR, horizon 384). 1. EVERY STANDARD ERROR ASSUMED INDEPENDENT SAMPLES. Triple-barrier labels started one per bar overlap by the label's lifespan, so n calls are worth ~n/L independent observations (Lopez de Prado, AFML ch. 4 - sample uniqueness). All three sqrt(p(1-p)/n) sites divided by the RAW count. The tell: the operating point's null-of-the-maximum gate is family-wise and should fire on ~5% of eras under the null. Measured fire rates - PAI 47/73 (64%), ConvLSTM 9/24, LSTM 8/21 (38%), CONV 4/62 (6%). CONV, the only model whose margin distribution admits few bins, sat on the null; the rest cleared a bar that was too low by ~sqrt(L). PAI's deployed threshold consequently alternated between the ENDS of its own range era to era (0.10 -> 0.88 -> 0.86 -> 0.66; coverage 16% <-> 73%). TripleBarrierLabel now records when each label became KNOWABLE - the first winning touch, or both stops, or the timeout - and the prebuild accumulates the mean. EffectiveSampleSize() feeds the operating point, the member deploy gate and the ensemble vote gate. Conservative by construction (n/L is an upper bound on the damage); gates get harder, never easier. 2. THE SCALE LADDER RAN AWAY, again. Horizon scales as swingMedian*sl*tp, and since 4d8cb08 reachability is measured OVER that horizon - so a wider rung buys itself the time that makes it look reachable. Same target -> horizon -> reach -> target loop the excursion window is kept short to avoid; fixing the window confusion reopened it through the other door. It walked 128 -> 256 -> 384 bars and stopped at q90, the widest rung there is, with every rung reading 39-48% against a 20% floor. A floor nothing fails selects nothing. Rungs whose required horizon exceeds BARRIER_HORIZON_MAX are now rejected - the same rule ReportGeometryExpectancyScan already applied. It was printing the shipped pair as CLAMPED and disqualified ('h384!') two lines under the deriver that chose it: two subsystems, one geometry, opposite verdicts. 3. THE RUNG SNAP DESTROYED THE RATIO IT WAS COMPARING. Both legs snapped independently to the coarse first-passage grid, re-rating each candidate: q90 4.86/9.71 -> 5.00/10.00 (2.00), q85 4.07/8.14 -> 5.00/10.00 (IDENTICAL measurement), q75 3.07/6.13 -> 4.00/6.50 (1.63 - a nearer target). So the ladder compared win shares taken at ratios from 1.63 to 2.17 and read the differences as scale. It is why the reach column came out non-monotone in width (q75 48.5% above q90 42.9%). The stop now snaps to its nearest rung in log space and the target follows the ratio off it; the pair actually measured is returned and logged, so a collision reads as a collision. Also: LadderWinShare guarded against the conditional (fractal) geometry path, which fills n from m_fracLegCount while leaving idxList empty - a latent out-of-bounds on a currently-dead path. New log lines: mean label lifespan and effective n on the label-cache line, the required-vs-available horizon per rung, and the grid pair the reconciliation actually measured (its tolerance now scales with the grid skew instead of a flat 5pp). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 01:12:05 -04:00
double LadderWinShare(const int &idxList[], int n, double sl, double tp,
double &effSl, double &effTp);
//--- Bars this (sl, tp) pair needs before its label means "target before stop" rather than
//--- "target before stop OR 384 bars, whichever comes first".
fix(labels): overlapping-label sample correction + horizon cap on the scale ladder Three defects, all surfaced by the 2026-08-17 SP500 H4 run that shipped stop 4.86 / target 9.71 (width 14.57*ATR, horizon 384). 1. EVERY STANDARD ERROR ASSUMED INDEPENDENT SAMPLES. Triple-barrier labels started one per bar overlap by the label's lifespan, so n calls are worth ~n/L independent observations (Lopez de Prado, AFML ch. 4 - sample uniqueness). All three sqrt(p(1-p)/n) sites divided by the RAW count. The tell: the operating point's null-of-the-maximum gate is family-wise and should fire on ~5% of eras under the null. Measured fire rates - PAI 47/73 (64%), ConvLSTM 9/24, LSTM 8/21 (38%), CONV 4/62 (6%). CONV, the only model whose margin distribution admits few bins, sat on the null; the rest cleared a bar that was too low by ~sqrt(L). PAI's deployed threshold consequently alternated between the ENDS of its own range era to era (0.10 -> 0.88 -> 0.86 -> 0.66; coverage 16% <-> 73%). TripleBarrierLabel now records when each label became KNOWABLE - the first winning touch, or both stops, or the timeout - and the prebuild accumulates the mean. EffectiveSampleSize() feeds the operating point, the member deploy gate and the ensemble vote gate. Conservative by construction (n/L is an upper bound on the damage); gates get harder, never easier. 2. THE SCALE LADDER RAN AWAY, again. Horizon scales as swingMedian*sl*tp, and since 4d8cb08 reachability is measured OVER that horizon - so a wider rung buys itself the time that makes it look reachable. Same target -> horizon -> reach -> target loop the excursion window is kept short to avoid; fixing the window confusion reopened it through the other door. It walked 128 -> 256 -> 384 bars and stopped at q90, the widest rung there is, with every rung reading 39-48% against a 20% floor. A floor nothing fails selects nothing. Rungs whose required horizon exceeds BARRIER_HORIZON_MAX are now rejected - the same rule ReportGeometryExpectancyScan already applied. It was printing the shipped pair as CLAMPED and disqualified ('h384!') two lines under the deriver that chose it: two subsystems, one geometry, opposite verdicts. 3. THE RUNG SNAP DESTROYED THE RATIO IT WAS COMPARING. Both legs snapped independently to the coarse first-passage grid, re-rating each candidate: q90 4.86/9.71 -> 5.00/10.00 (2.00), q85 4.07/8.14 -> 5.00/10.00 (IDENTICAL measurement), q75 3.07/6.13 -> 4.00/6.50 (1.63 - a nearer target). So the ladder compared win shares taken at ratios from 1.63 to 2.17 and read the differences as scale. It is why the reach column came out non-monotone in width (q75 48.5% above q90 42.9%). The stop now snaps to its nearest rung in log space and the target follows the ratio off it; the pair actually measured is returned and logged, so a collision reads as a collision. Also: LadderWinShare guarded against the conditional (fractal) geometry path, which fills n from m_fracLegCount while leaving idxList empty - a latent out-of-bounds on a currently-dead path. New log lines: mean label lifespan and effective n on the label-cache line, the required-vs-available horizon per rung, and the grid pair the reconciliation actually measured (its tolerance now scales with the grid skew instead of a flat 5pp). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 01:12:05 -04:00
int RequiredHorizonBars(double slMult, double tpMult);
//--- Bars it would actually GET: the above, clamped to [MIN, MAX] and snapped DOWN to the
//--- ladder. These are two different numbers and conflating them is its own trap - the ceiling
//--- clamp is what ReportGeometryExpectancyScan disqualifies with '!', but the snap-down
//--- truncates as well and is silent about it (a pair needing 317 bars is granted 256).
fix(labels): correct EffectiveSampleSize clamp order, share the horizon ladder, retract a false justification Self-review of 1540ba8 against the FULL 6,930-era log rather than the first three minutes of it. Three corrections. 1. EffectiveSampleSize() clamped in the wrong order. MathMax(2, MathMin(eff, rawN)) returns 2 when rawN is 1 - an effective sample LARGER than the raw one, shrinking the SE in exactly the direction the function exists to prevent. Floor first, cap at rawN last. 2. The horizon cap rejected on the CEILING only, and said so as though that made the label untruncated. It does not: the horizon ladder also snaps DOWN, so a pair needing 317 bars is granted 256 and is silently truncated without ever being flagged CLAMPED. Added SnapHorizonToLadder() / GrantedHorizonBars() and the scale ladder now reports "needs N gets M" per rung. Rejection stays on the ceiling alone - matching ReportGeometryExpectancyScan's '!' exactly, which was the point - because rejecting on the snap-down would select rungs for landing just above a ladder point rather than for anything about the market. ComputeBarrierHorizonBars' private copy of the ladder is gone; there is now one copy, which is the whole reason RequiredHorizonBars was factored out. 3. RETRACTED THE JUSTIFICATION IN 1540ba8's COMMENTS. That commit claimed the overlap correction was needed because the operating point's null-of-the- maximum gate fired on 47/73 Perceptron eras (64%) where a family-wise test should fire on ~5%. Those 73 fits were the first three minutes of a six-and-a-half-hour run. Over the full run: PAI 47/3214 = 1.5% HYB 30/1200 = 2.5% CONV 4/63 = 6.3% LSTM 75/915 = 8.2% All at or below the null. The gate from 7414570 is working as designed and PAI's 47 clears were a cold-start transient never repeated in 3,141 later fits; its threshold over the run's second half has sd 0.01. The overlap correction is still right - sqrt(p(1-p)/n) on overlapping labels is the wrong formula - but it fixes no observed failure, and it costs nothing today because no model is near the deploy line. WHAT THE FULL RUN DOES CONFIRM, unchanged: the geometry ran away exactly as described (2.00/6.00 h128 -> 3.49/6.99 h256 -> 4.86/9.71 h384, three passes, stopping at q90 because the quantile ladder ended), the label stayed long-skewed at Buy 42.9% / Sell 22.6%, and no checkpoint on any of the four models ever cleared the deployability floor. Pooled declustered win rates: PAI 31.70%, HYB 31.02%, LSTM 32.69%, CONV 31.57% - every one 4-6pp below the 37% always-long chance rate and 1-2.7pp below the 33.7% cost-adjusted break-even. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 07:30:19 -04:00
int SnapHorizonToLadder(int rawBars);
int GrantedHorizonBars(double slMult, double tpMult);
fix(labels): overlapping-label sample correction + horizon cap on the scale ladder Three defects, all surfaced by the 2026-08-17 SP500 H4 run that shipped stop 4.86 / target 9.71 (width 14.57*ATR, horizon 384). 1. EVERY STANDARD ERROR ASSUMED INDEPENDENT SAMPLES. Triple-barrier labels started one per bar overlap by the label's lifespan, so n calls are worth ~n/L independent observations (Lopez de Prado, AFML ch. 4 - sample uniqueness). All three sqrt(p(1-p)/n) sites divided by the RAW count. The tell: the operating point's null-of-the-maximum gate is family-wise and should fire on ~5% of eras under the null. Measured fire rates - PAI 47/73 (64%), ConvLSTM 9/24, LSTM 8/21 (38%), CONV 4/62 (6%). CONV, the only model whose margin distribution admits few bins, sat on the null; the rest cleared a bar that was too low by ~sqrt(L). PAI's deployed threshold consequently alternated between the ENDS of its own range era to era (0.10 -> 0.88 -> 0.86 -> 0.66; coverage 16% <-> 73%). TripleBarrierLabel now records when each label became KNOWABLE - the first winning touch, or both stops, or the timeout - and the prebuild accumulates the mean. EffectiveSampleSize() feeds the operating point, the member deploy gate and the ensemble vote gate. Conservative by construction (n/L is an upper bound on the damage); gates get harder, never easier. 2. THE SCALE LADDER RAN AWAY, again. Horizon scales as swingMedian*sl*tp, and since 4d8cb08 reachability is measured OVER that horizon - so a wider rung buys itself the time that makes it look reachable. Same target -> horizon -> reach -> target loop the excursion window is kept short to avoid; fixing the window confusion reopened it through the other door. It walked 128 -> 256 -> 384 bars and stopped at q90, the widest rung there is, with every rung reading 39-48% against a 20% floor. A floor nothing fails selects nothing. Rungs whose required horizon exceeds BARRIER_HORIZON_MAX are now rejected - the same rule ReportGeometryExpectancyScan already applied. It was printing the shipped pair as CLAMPED and disqualified ('h384!') two lines under the deriver that chose it: two subsystems, one geometry, opposite verdicts. 3. THE RUNG SNAP DESTROYED THE RATIO IT WAS COMPARING. Both legs snapped independently to the coarse first-passage grid, re-rating each candidate: q90 4.86/9.71 -> 5.00/10.00 (2.00), q85 4.07/8.14 -> 5.00/10.00 (IDENTICAL measurement), q75 3.07/6.13 -> 4.00/6.50 (1.63 - a nearer target). So the ladder compared win shares taken at ratios from 1.63 to 2.17 and read the differences as scale. It is why the reach column came out non-monotone in width (q75 48.5% above q90 42.9%). The stop now snaps to its nearest rung in log space and the target follows the ratio off it; the pair actually measured is returned and logged, so a collision reads as a collision. Also: LadderWinShare guarded against the conditional (fractal) geometry path, which fills n from m_fracLegCount while leaving idxList empty - a latent out-of-bounds on a currently-dead path. New log lines: mean label lifespan and effective n on the label-cache line, the required-vs-available horizon per rung, and the grid pair the reconciliation actually measured (its tolerance now scales with the grid skew instead of a flat 5pp). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 01:12:05 -04:00
//--- Independent-observation count behind `rawN` overlapping triple-barrier labels. See
//--- m_lastLabelLifespan for the measurement and for what an uncorrected n did to the operating point.
fix(topology): the capacity budget counted overlapping bars as independent examples EstimatedInSampleBars() returned raw bars (11372 on SP500 H4) and every derived capacity decision spent that: first-layer width, conv filters, LSTM hidden size. But triple-barrier labels overlap - mean lifespan 9.4 bars - so the label cache line on the same run already reports those bars are worth ~1210 independent observations. Sizing a network against RAW bars while grading it against EFFECTIVE ones is two subsystems disagreeing about one sample, and it disagreed in the dangerous direction because the capacity side was the optimistic one: the warning's "roughly 1.1 weights per training bar" is nearer 11 per independent observation. EffectiveSampleSize() has existed since 2026-08-17 and is applied at eight sites, all of them statistics. This adds the ninth, in the one place that decides how many parameters get fitted. Applied inside EstimatedInSampleBars() rather than at the call sites, because that function exists precisely so the three stages spend one budget. SELF-ENABLING AND THEREFORE INERT WHERE IT MATTERS MOST, which is why this is two changes and not one. MeanLabelLifespan() is 1.0 until a label cache has measured something, so on a model's first build - before any label exists - the deflation is correctly the identity: an unmeasured overlap must not invent a shrink. A fresh attach constructs a fresh object, so its counters are zero too; only a mid-session weights reset carries real evidence into a rebuild. That is deliberately safe (no attach can now re-derive a narrower topology and discard trained weights) but it would have left the first build - the case you most want the truth for - quoting the flattering figure. So ReportDetectability now restates capacity against the effective sample at the first moment L is real, for the topology already pinned. It re-sizes nothing; it reports what was bought. Placed ABOVE that function's break-even guard on purpose - a degenerate geometry is exactly when you want to know the net is over-parameterised, and "it only fires for sane configs" is how the 2026-08-18 IS-error stop managed never to fire at all. The warning also names its basis now (independent observations and L, or an explicit "overlap NOT YET MEASURED, this is an UPPER BOUND"), so a flattering number can never again read as a measured one. Also factors FirstLayerFanIn() out of ComputeFirstLayerWidth so the capacity REPORT charges for exactly what the capacity DECISION charged for - same reason RequiredHorizonBars was factored out after the 2026-08-17 divergence - and makes MeanLabelLifespan()/EffectiveSampleSize() const so the const budget path can call them. Verified: no recursion (EstimatedInSampleBars -> EffectiveSampleSize -> EstimatedInSampleBarsRaw, which computes from Bars() alone); both new StringFormat sites hand-counted (basis 3/3 and 1/1, CAPACITY 10 specifiers / 10 arguments). NOT COMPILED - user compiles in MetaEditor.
2026-08-19 18:43:48 -04:00
double EffectiveSampleSize(double rawN) const;
fix(labels): overlapping-label sample correction + horizon cap on the scale ladder Three defects, all surfaced by the 2026-08-17 SP500 H4 run that shipped stop 4.86 / target 9.71 (width 14.57*ATR, horizon 384). 1. EVERY STANDARD ERROR ASSUMED INDEPENDENT SAMPLES. Triple-barrier labels started one per bar overlap by the label's lifespan, so n calls are worth ~n/L independent observations (Lopez de Prado, AFML ch. 4 - sample uniqueness). All three sqrt(p(1-p)/n) sites divided by the RAW count. The tell: the operating point's null-of-the-maximum gate is family-wise and should fire on ~5% of eras under the null. Measured fire rates - PAI 47/73 (64%), ConvLSTM 9/24, LSTM 8/21 (38%), CONV 4/62 (6%). CONV, the only model whose margin distribution admits few bins, sat on the null; the rest cleared a bar that was too low by ~sqrt(L). PAI's deployed threshold consequently alternated between the ENDS of its own range era to era (0.10 -> 0.88 -> 0.86 -> 0.66; coverage 16% <-> 73%). TripleBarrierLabel now records when each label became KNOWABLE - the first winning touch, or both stops, or the timeout - and the prebuild accumulates the mean. EffectiveSampleSize() feeds the operating point, the member deploy gate and the ensemble vote gate. Conservative by construction (n/L is an upper bound on the damage); gates get harder, never easier. 2. THE SCALE LADDER RAN AWAY, again. Horizon scales as swingMedian*sl*tp, and since 4d8cb08 reachability is measured OVER that horizon - so a wider rung buys itself the time that makes it look reachable. Same target -> horizon -> reach -> target loop the excursion window is kept short to avoid; fixing the window confusion reopened it through the other door. It walked 128 -> 256 -> 384 bars and stopped at q90, the widest rung there is, with every rung reading 39-48% against a 20% floor. A floor nothing fails selects nothing. Rungs whose required horizon exceeds BARRIER_HORIZON_MAX are now rejected - the same rule ReportGeometryExpectancyScan already applied. It was printing the shipped pair as CLAMPED and disqualified ('h384!') two lines under the deriver that chose it: two subsystems, one geometry, opposite verdicts. 3. THE RUNG SNAP DESTROYED THE RATIO IT WAS COMPARING. Both legs snapped independently to the coarse first-passage grid, re-rating each candidate: q90 4.86/9.71 -> 5.00/10.00 (2.00), q85 4.07/8.14 -> 5.00/10.00 (IDENTICAL measurement), q75 3.07/6.13 -> 4.00/6.50 (1.63 - a nearer target). So the ladder compared win shares taken at ratios from 1.63 to 2.17 and read the differences as scale. It is why the reach column came out non-monotone in width (q75 48.5% above q90 42.9%). The stop now snaps to its nearest rung in log space and the target follows the ratio off it; the pair actually measured is returned and logged, so a collision reads as a collision. Also: LadderWinShare guarded against the conditional (fractal) geometry path, which fills n from m_fracLegCount while leaving idxList empty - a latent out-of-bounds on a currently-dead path. New log lines: mean label lifespan and effective n on the label-cache line, the required-vs-available horizon per rung, and the grid pair the reconciliation actually measured (its tolerance now scales with the grid skew instead of a flat 5pp). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 01:12:05 -04:00
//--- Mean bars-to-resolution over the label cache, or 1.0 before anything has been measured (which
//--- makes EffectiveSampleSize the identity, i.e. the old behaviour, rather than a guess).
fix(topology): the capacity budget counted overlapping bars as independent examples EstimatedInSampleBars() returned raw bars (11372 on SP500 H4) and every derived capacity decision spent that: first-layer width, conv filters, LSTM hidden size. But triple-barrier labels overlap - mean lifespan 9.4 bars - so the label cache line on the same run already reports those bars are worth ~1210 independent observations. Sizing a network against RAW bars while grading it against EFFECTIVE ones is two subsystems disagreeing about one sample, and it disagreed in the dangerous direction because the capacity side was the optimistic one: the warning's "roughly 1.1 weights per training bar" is nearer 11 per independent observation. EffectiveSampleSize() has existed since 2026-08-17 and is applied at eight sites, all of them statistics. This adds the ninth, in the one place that decides how many parameters get fitted. Applied inside EstimatedInSampleBars() rather than at the call sites, because that function exists precisely so the three stages spend one budget. SELF-ENABLING AND THEREFORE INERT WHERE IT MATTERS MOST, which is why this is two changes and not one. MeanLabelLifespan() is 1.0 until a label cache has measured something, so on a model's first build - before any label exists - the deflation is correctly the identity: an unmeasured overlap must not invent a shrink. A fresh attach constructs a fresh object, so its counters are zero too; only a mid-session weights reset carries real evidence into a rebuild. That is deliberately safe (no attach can now re-derive a narrower topology and discard trained weights) but it would have left the first build - the case you most want the truth for - quoting the flattering figure. So ReportDetectability now restates capacity against the effective sample at the first moment L is real, for the topology already pinned. It re-sizes nothing; it reports what was bought. Placed ABOVE that function's break-even guard on purpose - a degenerate geometry is exactly when you want to know the net is over-parameterised, and "it only fires for sane configs" is how the 2026-08-18 IS-error stop managed never to fire at all. The warning also names its basis now (independent observations and L, or an explicit "overlap NOT YET MEASURED, this is an UPPER BOUND"), so a flattering number can never again read as a measured one. Also factors FirstLayerFanIn() out of ComputeFirstLayerWidth so the capacity REPORT charges for exactly what the capacity DECISION charged for - same reason RequiredHorizonBars was factored out after the 2026-08-17 divergence - and makes MeanLabelLifespan()/EffectiveSampleSize() const so the const budget path can call them. Verified: no recursion (EstimatedInSampleBars -> EffectiveSampleSize -> EstimatedInSampleBarsRaw, which computes from Bars() alone); both new StringFormat sites hand-counted (basis 3/3 and 1/1, CAPACITY 10 specifiers / 10 arguments). NOT COMPILED - user compiles in MetaEditor.
2026-08-19 18:43:48 -04:00
double MeanLabelLifespan(void) const;
//--- Last era's deploy-gate arithmetic, published purely so the era line can state the bar
//--- rather than leave it implicit. -1 = not computed this era.
feat(measurement): fix zero-skill denominator, publish the deploy bar, measure lifespan per rung, add a MEASURE scale objective The last run could not have demonstrated an edge either way, and nothing in the log said so. Four changes so it does. 1. THE ZERO-SKILL LINE DIVIDED BY THE WRONG DENOMINATOR. m_oosWinLongTotal resets every era; m_oosSamples only resets on a full model reset. So 'always-long %' decayed as ~1/era: a run whose true rate is 37% printed 1.2% at era 33 and 0.0% at era 2219. This is the SAME bug already found and fixed for logBuyPredPct thirty lines above ('era-15 Buy:2% that was really ~30%'), left in the one line whose whole job is to be the reference every other number is read against. Correct at era 1, wrong everywhere after - including the '62% zero-skill' figure in the 2026-08-16 notes. Now per-era, and always-short is finally readable. 2. THE DEPLOY GATE STATES ITS OWN BAR. 'edge -1pp' era after era cannot separate 'short by a hair' from 'short by an amount no strategy could cover'. The era line now prints the required win rate, the SE, the effective n and the lifespan it was deflated by; above 100% it says UNREACHABLE. At 4,738 OOS bars and L=75.6 there are ~63 independent observations, putting the bar near 66% at typical coverage. 3. LIFESPAN MEASURED PER RUNG. The first-passage cache already stores touch ages at every ladder level, so each candidate geometry's resolution time is readable without training on it - L-vs-width becomes a measurement across the whole ladder in ONE run rather than a second chart. Each rung reports L, n_eff, min provable edge and min provable EV. 4. SCALE OBJECTIVE IS PHASE-AWARE, defaulting to MEASURE. Width and detectability are opposed: labels overlap by L, L grows like m*k = width^2 at fixed ratio, so min provable EV ~ width^2 while the cost saving from width is only linear. Doubling width quadruples the smallest EV you can prove. DEPLOY (widest that clears reachability) is right once an edge is known; MEASURE (narrowest that keeps round-trip spread under BARRIER_MAX_COST_FRACTION_PCT) is right while it still has to be shown. The direction does not depend on the exponent, and item 3 makes the exponent checkable. Fixed in review: m_lastRungLifespan is cleared on every LadderWinShare entry or a rejected rung reports the previous rung's lifespan as its own; per-rung detectability is labelled IS-sample based (the deriver may not see the holdout), so absolute figures are optimistic while the ranking is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:37:05 -04:00
double m_lastEdgeFloorPct;
double m_lastPrecSE;
double m_lastEffN;
//--- Mean bars-to-resolution at the rung LadderWinShare() last measured - a candidate geometry's
//--- lifespan, readable without training on it. See the scale ladder's detectability column.
double m_lastRungLifespan;
feat(gate): cross-instrument pooled certification The deploy bottleneck is CERTIFICATION, not training. A 4,738-bar OOS window at L=75.6 holds ~63 independent observations; certifying a 3pp edge at 2 sigma needs ~1,036. More bars of the same symbol barely help - they overlap. Other symbols do not. WHAT POOLS. Not win rates: symbols have different derived geometries, different break-evens and different drifts, so averaging raw rates across them is meaningless. What pools is each symbol's EXCESS OVER ITS OWN CHANCE RATE, combined by inverse-variance weighting (fixed-effects meta-analysis). Each symbol keeps its own model, geometry and chance rate; only the evidence is combined. THE CORRELATION PROBLEM, bracketed rather than assumed away. SP500 and NAS100 are ~0.9 correlated and pooling them as independent inflates the evidence. Nothing here can measure that without sharing return series, so instead of guessing a correction the gate reports both ends: SE_INDEP = sqrt(1/SUM(1/var_i)) all members independent SE_CORR = SUM(w_i * sqrt(var_i)) all members perfectly correlated The truth is always between. THE GATE USES SE_CORR, so a pass cannot be an artifact of correlated instruments - that bound already assumes the worst. The ratio is logged as the diversification credit the gate declines to claim, so the cost of that conservatism is visible instead of hidden. SCOPE, deliberately limited: the pooled result is REPORTED, never folded into tradeableOK. The local gate certifies the model that actually trades this symbol; the pool answers the different question of whether the strategy has an edge at all. Letting a cross-symbol result license a local deploy would ship a model that never cleared its own bar - so it cannot. Mechanics: one file per instrument (no concurrent-write path to get wrong), every FileOpen carrying FILE_SHARE_READ|FILE_SHARE_WRITE, records skipped rather than reinterpreted on a version mismatch, 12h staleness cutoff so a stopped chart cannot vote, and pooling refused below 3 instruments. Poolability requires matching timeframe and ratio; differing SYMBOL is the entire point. Publishing is unconditional - a pool that only hears from winners is a selection effect, not a meta-analysis. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:45:37 -04:00
//--- CROSS-INSTRUMENT CERTIFICATION - see AIBase\PooledGate.mqh for why the deploy bottleneck is
//--- certification rather than training, and why only the evidence pools while each symbol keeps its
//--- own model, geometry and chance rate.
void PublishPoolRecord(double chancePct, double winPct, double effN);
int ReadPooledEvidence(double &pooledExcessPp, double &seIndep, double &seCorr,
string &detail);
bool PooledGatePasses(string &report);
bool m_poolWriteWarned; // one-shot: a pool that cannot be written must say so
//--- Last era's pooled verdict, cached for the era line. Letting a cross-symbol result license a
//--- local deploy would ship a model that never cleared its own bar.
feat(gate): cross-instrument pooled certification The deploy bottleneck is CERTIFICATION, not training. A 4,738-bar OOS window at L=75.6 holds ~63 independent observations; certifying a 3pp edge at 2 sigma needs ~1,036. More bars of the same symbol barely help - they overlap. Other symbols do not. WHAT POOLS. Not win rates: symbols have different derived geometries, different break-evens and different drifts, so averaging raw rates across them is meaningless. What pools is each symbol's EXCESS OVER ITS OWN CHANCE RATE, combined by inverse-variance weighting (fixed-effects meta-analysis). Each symbol keeps its own model, geometry and chance rate; only the evidence is combined. THE CORRELATION PROBLEM, bracketed rather than assumed away. SP500 and NAS100 are ~0.9 correlated and pooling them as independent inflates the evidence. Nothing here can measure that without sharing return series, so instead of guessing a correction the gate reports both ends: SE_INDEP = sqrt(1/SUM(1/var_i)) all members independent SE_CORR = SUM(w_i * sqrt(var_i)) all members perfectly correlated The truth is always between. THE GATE USES SE_CORR, so a pass cannot be an artifact of correlated instruments - that bound already assumes the worst. The ratio is logged as the diversification credit the gate declines to claim, so the cost of that conservatism is visible instead of hidden. SCOPE, deliberately limited: the pooled result is REPORTED, never folded into tradeableOK. The local gate certifies the model that actually trades this symbol; the pool answers the different question of whether the strategy has an edge at all. Letting a cross-symbol result license a local deploy would ship a model that never cleared its own bar - so it cannot. Mechanics: one file per instrument (no concurrent-write path to get wrong), every FileOpen carrying FILE_SHARE_READ|FILE_SHARE_WRITE, records skipped rather than reinterpreted on a version mismatch, 12h staleness cutoff so a stopped chart cannot vote, and pooling refused below 3 instruments. Poolability requires matching timeframe and ratio; differing SYMBOL is the entire point. Publishing is unconditional - a pool that only hears from winners is a selection effect, not a meta-analysis. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:45:37 -04:00
bool m_lastPoolPasses;
string m_lastPoolReport;
//--- Break-even WITH the spread, which is the bar a model actually has to clear. A win nets (TP
//--- - spread), a loss costs (SL + spread). Falls back to frictionless when m_spreadAtr is unset
//--- (a loaded model that has not re-derived).
fix(geometry): the target was small BY CONSTRUCTION - ratio is now policy, scale is measured, ladder ceiling removed The derivation read the stop from q75 of ADVERSE travel and the target from q50 of FAVOURABLE travel. Over one horizon those distributions are broadly the same shape, so q75 > q50 MECHANICALLY - the target came out smaller than the stop no matter what the market did. SP500 H4 shipped stop 3.07 / target 1.70: a 0.55:1 payoff needing 64.3%. That was never a measurement, it was two mismatched constants. The reachability line printed beside it - "target on 50.0% of bars, stop on 25.0%" - is exactly 1-q50 and 1-q75. Tautological. It cannot disconfirm anything, and it read as validation. WIDTH AND RATIO ARE INDEPENDENT AND ONLY ONE PAYS. EV = edge x width; ratio is EV-neutral (a driftless walk reaches +m before -k with probability k/(k+m), which IS break-even). Width is what buys cost efficiency: the spread is a fixed 0.047*ATR here, so the shipped 4.77*ATR width paid it 21 times per unit of travel. So: RATIO = policy. BARRIER_TARGET_RR = 2.0 (user's 1:2). Break-even 33.3%. SCALE = measured. The stop quantile is chosen from a ladder, WIDEST FIRST, taking the first rung whose implied 2x target is still reached often enough to be a trainable class. That last clause is the difference from the min-reward:risk raise removed in 2026-08-09, which forced target = 2 x stop with NO reachability test, landed on 6.66*ATR reachable on 3.3% of bars, and trained the model to predict something that essentially never happened. Same ratio; the scale now retreats until the data says the target is attainable. Every rung is logged. LADDER CEILING REMOVED. BARRIER_LADDER stopped at 5.00 and the expectancy scan's "best resolvable pair on width alone" came back as stop 5.05 / target 4.95 - it pinned to the top rung. A recommendation landing exactly on the edge of its own search space is a boundary, not a finding: it cannot tell "5 ATR is optimal" from "5 ATR is all we allowed". Extended to 20*ATR (8 -> 14 rungs). Nothing else needs editing - every consumer is parameterised by BARRIER_LADDER_COUNT - and the horizon constraints (decided >= 60%, reachability floor) now bind instead of a constant. THE SCAN COULD NOT SEE THE SHIPPED GEOMETRY. ReportBarrierGeometryScan looked the configured pair up in its integer grid, and DeriveBarrierGeometry produces CONTINUOUS multiples (3.07/1.70) that can never equal a grid point - so cfgExcess stayed at its -1.0 sentinel and the report printed "configured 3:2 scores -1.00000", which reads as a catastrophic score and actually means "never evaluated". Worse, the grid skipped target<stop entirely because it "inverts the trade's whole premise" - while the derivation was shipping exactly that. The incumbent is now always scored as a peer (never crowned; it is already in force and is not an enum pairing the scan could adopt). BREAK-EVEN NOW INCLUDES THE SPREAD. Every report quoted the frictionless SL/(SL+TP). On SP500 H4 that read 64.3% while the MEASURED zero-skill rate was 62.1% - a 2.2pp gap that IS the cost, and that made every model look 2.2pp better than it was. CostAdjustedBreakEvenPct() prices a win at (TP - spread) and a loss at (SL + spread), matching the expectancy scan's convention exactly so the two reports cannot disagree. It also feeds FitDirConfThreshold, which is the correctness half: the operating point subtracts break-even from precision, so the frictionless figure made every candidate threshold look better by the width of the spread - 2.2pp against a measured edge of 2.3pp, i.e. very nearly all of it. Era line now carries both: "break-even 64.3% frictionless, 66.6% AFTER SPREAD". Forces a full relabel and retrain. Requested. NOT COMPILED - user compiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 00:34:32 -04:00
double CostAdjustedBreakEvenPct(void);
//--- Spread in ATR units, averaged over the IS bars - measured in ReportGeometryExpectancyScan, which
//--- is the only place with both the ATR series and the label cache in hand. 0 = not yet measured.
double m_spreadAtr;
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- Median confirmed-ZigZag-leg length over the training window, snapped to the horizon ladder.
int ComputeBarrierHorizonBars(int bars);
//--- Resolves m_barrierHorizonBars exactly once per process, from live buffers. Needed on BOTH
//--- paths, which is the whole reason it is not simply inlined in the prebuild: a DEPLOYED model
//--- never enters Train(), so it never reaches StartLabelCachePrebuild() - yet OnlineLearnStep()
//--- reads the horizon as its confirmation delay.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
void EnsureBarrierHorizon(int bars);
bool m_barrierHorizonResolved;
//--- full per-bar INPUT feature vector cache (everything BufferTempData() computes: ATR-
//--- normalized OHLC, time-of-day encoding, volume delta, AD indicator buffers, ...).
double m_featureCache[];
fix: the trailing incumbent read the future across eras; cold AD blocks cached zeros as truth Three findings from the 2026-08-11 audit: 1. The excursion head's trailing-quantile ring was deliberately never cleared between eras ("a rolling estimate of the market, not of the era") - but pass 3 re-walks the SAME OOS window every era, so at each walk's restart the ring still held the outcome masks of the newest OOS bars from the previous walk: the chronological FUTURE of the bars about to be scored. For the first ~window+horizon pushes of every era the "trailing" incumbent was partly a leading one - conservative for the gate (an informed incumbent is a harder hurdle) but exactly the self-made-artifact class 06d4785 hunts. The ring now clears at era-score reset; the warm-up bars simply don't score the trail race, which the m_excTrailN gating already accounts for. 2. skillTrail compared the head's FULL-block Brier (pro-rated by coverage) against the incumbent's subset sum - valid only if head skill is uniform across the OOS walk, while the trail-scored subset systematically excludes each era's warm-up bars. The audit also found m_excBrierHeadD/BaseD/ m_excOosHitsD declared, zeroed and never accumulated (dead since e2c9593 made every scored bar disjoint). The dead trio is replaced by m_excBrierHeadT: the head's Brier accumulated only on the bars the warm incumbent also scored, so the race now compares both predictors on an identical bar set. 3. The AD/Wyckoff feature blocks read GetData with no EMPTY_VALUE guard; a cold (still-calculating) indicator returns EMPTY_VALUE everywhere, the sanitize loop rewrote that to 0.0, and the bar SUCCEEDED - so BufferTempData cached an all-zero Wyckoff block as a success for the whole bar frame: the one path the f6150ee only-cache-successes rule cannot see, because it never fails (the ba13eef class, arriving through values that never fail; a resumed model's era-0 prebuild starts milliseconds after OnInit). ADIndicatorCold() probes the NEWEST bar - EMPTY_VALUE there means async warm-up (transient reject, retried), while deep bars beyond the buffered depth keep the sanitize loop's neutral-fill so degraded history still trains. Also fixed m_featureCacheValid's declaration comment, which still described the pre-f6150ee cached-miss semantics. Compile: 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 18:20:24 -04:00
bool m_featureCacheHasValue[]; // true once idx has a CACHED SUCCESS (f6150ee: only
// successes are ever cached - a miss is never stored,
// in any form; see BufferTempData's comment)
bool m_featureCacheValid[]; // paired flag, always true when HasValue is true -
// kept for the (currently unreachable) cached-miss
// shape so the cache layout survives f6150ee
fix: a resumed model cached a cold ATR as permanent, so it never trained BufferTempData cached EVERY failure - m_featureCacheHasValue[idx]=true with m_featureCacheValid[idx]=false - and the cache never re-tries a miss. So a single feature read taken before the terminal had finished calculating the indicator buffers marked those bars unusable for the rest of the process, even though the data arrived milliseconds later. MT5 fills an indicator's buffers asynchronously after the handle is created, and a cold ATR returns 0 for EVERY index, not just its warm-up tail. BufferTempDataCompute rejects a bar with no ATR (correctly - the price features would be meaningless), so the whole window failed, and the whole cache was poisoned. Only resumed models were hit, because only they read features that early. Topology.mqh sets m_warmupPassesRemaining = netLoaded ? 0 : 3: a fresh start sits through three separately-scheduled Train() calls before anything touches a feature, which is exactly what those passes are for. A resumed one skips them and TuneIndicatorsAndTrain drives StartLabelCachePrebuild and the MI report from the first chart event. Its rationale - "a restart already has a proven-synced history" - holds for HISTORY and not for INDICATORS, which are recreated every process start. Downstream: BuildFeatureWindow failed on every bar of every era, so add_loop never went true, so pass 2, pass 3, the era counter and the checkpoint were all skipped and pass 1 swept 0->100% forever. The "0 samples" MI report line at startup was the same failure, four seconds earlier, already visible in the log. - a miss is now cached only when it is PERMANENT; the two "not ready yet" guards mark m_featureFailTransient and are recomputed on the next visit. Steady-state cost is ~ind_Periods bars per era, not 54k. - an era that discards itself now drops the feature cache before restarting, so any remaining cause of this state self-heals instead of looping. Deleting the .nnw "fixed" this only by turning the model back into a fresh one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:41:31 -04:00
//--- Set by BufferTempDataCompute when it rejected a bar because the data had not ARRIVED yet
//--- (price buffer EMPTY_VALUE, or an ATR the terminal has not finished calculating) as opposed
//--- to the bar being genuinely unusable.
fix: a resumed model cached a cold ATR as permanent, so it never trained BufferTempData cached EVERY failure - m_featureCacheHasValue[idx]=true with m_featureCacheValid[idx]=false - and the cache never re-tries a miss. So a single feature read taken before the terminal had finished calculating the indicator buffers marked those bars unusable for the rest of the process, even though the data arrived milliseconds later. MT5 fills an indicator's buffers asynchronously after the handle is created, and a cold ATR returns 0 for EVERY index, not just its warm-up tail. BufferTempDataCompute rejects a bar with no ATR (correctly - the price features would be meaningless), so the whole window failed, and the whole cache was poisoned. Only resumed models were hit, because only they read features that early. Topology.mqh sets m_warmupPassesRemaining = netLoaded ? 0 : 3: a fresh start sits through three separately-scheduled Train() calls before anything touches a feature, which is exactly what those passes are for. A resumed one skips them and TuneIndicatorsAndTrain drives StartLabelCachePrebuild and the MI report from the first chart event. Its rationale - "a restart already has a proven-synced history" - holds for HISTORY and not for INDICATORS, which are recreated every process start. Downstream: BuildFeatureWindow failed on every bar of every era, so add_loop never went true, so pass 2, pass 3, the era counter and the checkpoint were all skipped and pass 1 swept 0->100% forever. The "0 samples" MI report line at startup was the same failure, four seconds earlier, already visible in the log. - a miss is now cached only when it is PERMANENT; the two "not ready yet" guards mark m_featureFailTransient and are recomputed on the next visit. Steady-state cost is ~ind_Periods bars per era, not 54k. - an era that discards itself now drops the feature cache before restarting, so any remaining cause of this state self-heals instead of looping. Deleting the .nnw "fixed" this only by turning the model back into a fresh one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:41:31 -04:00
bool m_featureFailTransient;
//--- WHICH BLOCK rejected the bar, and at which series index.
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
string m_featureFailBlock;
int m_featureFailIdx;
diag: name the cause when every feature window fails, and enforce the width contract Era 0 stalls with "NOT ONE of 54681 scanned bars produced a usable feature window, windows ok=0 failed=54681" and nothing else. That line reads identically for a cold ATR, a conditionally-missing optional feature block and an out-of-range index, so it cannot be diagnosed without one restart per hypothesis. Two changes: 1. WIDTH CONTRACT in BufferTempData. Every enabled block must emit exactly m_neuronsCount values on EVERY bar. A block that emits its values on some bars and skips them on others (indicator, panel or series unavailable for that bar) does not merely shorten the window - it SHIFTS every feature after it into the wrong slot, and the net then trains on silently misaligned inputs that still look like a valid window to everything downstream. Now rejected, rolled back and reported once, naming the optional blocks (XA / SPR / swing context) as the ones carrying an availability test. Worth having independently of the current stall. 2. BuildFeatureWindow records WHICH lookback slot rejected and how much of the window was assembled, and the pass-1 stall report renders it: "slot 0 of 20 REJECTED (window had 0 of 760)" is an indicator warm-up or history-edge read; "every lookback bar ACCEPTED and the window was still short: 640 of 760" is a missing 6-value block. No behaviour change on a healthy run: the width check is an equality that already holds, and the diagnostics render only inside the total-failure branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 10:31:56 -04:00
//--- Why the LAST BuildFeatureWindow failed, so the pass-1 stall report can name a cause instead of
//--- a count. Slot = which lookback position rejected (-1 = none did and the window was still
//--- short); Total = how many values had been assembled when it gave up.
int m_windowFailSlot;
int m_windowFailTotal;
bool m_featureWidthWarned; // one-shot: the width contract is a structural fault
fix(altdata): median-fill instead of zero-fill, and a one-shot feature-vector autopsy ALT-DATA AUDIT. The files themselves are healthy - all six symbols, 6,073 daily rows, 2010-01-01 to 2026-08-17, no constant or degenerate columns, sane tails (mac_cpi/mac_unemp flat ~47d is monthly data behaving correctly). The problem is not the data, it is what happens where the data ISN'T. CAltDataPanel::Features() returned an all-ZERO vector for any bar older than the file's first row, and left blank cells at 0 too. Both were deliberate ('the block is additive context and must degrade, never reject the bar') and that reasoning holds for the CHANGE columns - but half these features are LEVELS: vix, ivol, mac_y10, mac_cpi, mac_unemp, eia_util. For a level, 0 is not a missing reading, it is an impossible one far outside the series' range. VIX does not visit zero. And the spike lands in exactly the wrong place. Every alt file starts 2010-01-01 while the charts run far deeper - USDJPY H4 reaches ~1994, roughly HALF its history - so 'alt block is all zeros' is precisely the predicate 'this bar is older than 2010'. The IS/OOS split is chronological, so that predicate covers ~half of IS and none of OOS: an in-sample feature guaranteed to be useless out-of-sample, and a bimodal input for the first BatchNorm to normalise. Not a lookahead leak - a distribution corruption, which is quieter and was never reported anywhere. Now filled with the column MEDIAN over the covered range. A constant cannot leak whatever its source - it takes the same value on every pre-coverage bar, so it carries no information about which of those bars won - which is what makes a median computed over later data legitimate here. Median not mean because the series are skewed. Blank cells get the same treatment (eia_stk_idx1y alone has 181 blanks in 6,073 rows) and the count is now logged at load. THE BACKOFF WAS ALREADY THERE AND WAS DEAD. Training.mqh arms m_coldSweepTick on m_featureFailTransient, but only the open/ATR guards ever set that flag, so f0cf659's cold ADMovingAverage looked PERMANENT and the sweep re-ran at full speed forever. Setting the flag in the indicator guards revives the mechanism that was already designed for this; no second backoff was needed and the one I first wrote has been removed in favour of it. SELF-HEALING, as asked. ReportFeatureHealth() runs once, the first time pass 1 produces usable windows, samples ~400 bars spread across the whole training range and names every feature slot that is CONSTANT or mostly-zero, tagging alt-block slots as alt[i]. Both of today's failures were the same shape - a block silently produces nothing while every downstream number stays plausible - and neither an accuracy figure nor a model can tell 'this feature is always 0' from 'this feature is genuinely 0 here'. Evenly spaced sampling so a block that dies only in deep history is caught as surely as one dead everywhere. A report, not a gate: a rare-flag feature can be legitimately constant, and refusing to train would turn a diagnostic into an outage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 12:53:51 -04:00
//--- One-shot feature-vector autopsy - see ReportFeatureHealth() for the two silent 2026-08-17
//--- failures it exists to catch. Runs the first time pass 1 produces usable windows.
void ReportFeatureHealth(int bars);
bool m_featureHealthReported;
bool BufferTempDataCompute(int idx);
//--- Nearest confirmed (non-repainting) ZigZag pivot at or after fromIdx - see this method's
//--- definition comment and m_useSwingContext's declaration comment for the repainting-embargo
//--- rationale callers must apply to fromIdx before calling this.
bool FindConfirmedZigZagPivot(int fromIdx, int &pivotIdx, double &pivotPrice, bool &pivotIsLow);
bool EnsureBarCachesCapacity(int bars);
//--- Eager label-cache pre-build + true-label tally, run once per fresh start (see
//--- m_warmupPassesRemaining) BEFORE era 0's real training loop begins.
bool m_labelCachePrebuilt; // true once the one-time pre-scan has completed
bool m_labelPrebuildActive; // true while a chunked pre-scan is in progress
bool m_prebuildSeedPending; // true: era 0's era-start reset must NOT stomp the
// prebuild-seeded m_prevEraTrue* counts with the
// still-empty live tally (see Train()'s era-start block)
int m_labelPrebuildBars;
int m_labelPrebuildOosCutoff;
int m_labelPrebuildIndex;
int m_labelPrebuildBuyCount;
int m_labelPrebuildSellCount;
int m_labelPrebuildNeutralCount;
void StartLabelCachePrebuild(void);
void AdvanceLabelCachePrebuild(void);
//--- Evaluation-only continual-learning OOS simulation: once the core model converges, a CLONE
//--- of its weights (never the production Net itself) walks forward through the OOS window bar-
//--- by-bar, scoring each bar with its current weights THEN learning from it - simulating how
//--- the model would adapt in live/forward trading.
CNet *m_simOosNet; // NULL when no simulation is active
bool m_simOosRunActive;
int m_simOosCutoff; // oosCutoff snapshot from the run that converged
int m_simOosBarIndex; // resume point, m_simOosCutoff-1 down to 0
double m_simOosForecast; // smoothed accuracy - separate from dOosForecast
int m_simOosSamples;
void StartOosContinualSimulation(int bars, int oosCutoff);
void AdvanceOosSimulationChunk(void);
//--- ONE-SHOT pattern-database backfill (user request 2026-08-16): "the DB needs to be filled
//--- during training so I do not have to run a backtest before deploying to live trading".
bool m_dbBackfillActive;
bool m_dbBackfillDone; // one-shot per deployment - never re-armed by a later call
int m_dbBackfillIndex; // resume point, descends to 2 (mirrors pass 3's m_oosScoreIndex)
int m_dbBackfillStartIndex;
fix(gate): move the ranking slice to the OLD end - it walled off the recent chart NOT COMPILED - user compiles. User: "there is quite some trading going on, but absolutely nothing on the recent area of the chart, like there is a hard wall starting around november 2025." That wall is 7caf2f6's ranking slice, and it was placed at the wrong end. Chart arrows are only ever drawn on bars pass 3 GRADES, and the slice reserved the NEWEST 20% of the OOS window plus a label-horizon purge. At the live sizing - ~4,860 OOS bars, 128-bar horizon - that is ~1,100 H4 bars withheld from grading, about ten months back from today, exactly where the wall appears. The invisible cost was worse than the visible one: it handed the deploy gate the OLDEST 80% of the OOS window and withheld the most recent regime from the single decision that has to generalise forward. Both fixed by putting the reserve at the oldest end instead: [0, oosScoreHi) OOS - graded by pass 3 (NEWEST, arrows restored) [oosScoreHi, rankLo) purge - one label horizon [rankLo, oosCutoff) RANKING - backfill only, graded by nobody [oosCutoff, calibLo) purge [calibLo, calibHi) CALIBRATION ... IS Of the three consumers competing for those bars, recency is worth least to the ranking: it is an ORDERING of confidence tiers, far less regime-sensitive than an absolute win rate, while the gate's power and the operator's read of the chart both want the newest data. The slice keeps every property that made it worth carving - never graded, never selected on, never seen by the gate, purged on both sides - so the backfilled rows are still honestly out-of-sample. RankSliceHiIndex is replaced by RankSliceLoIndex + OosScoreHiIndex; pass 3 now excludes the slice at the TOP of its walk and descends to 2 as it always did. The backfill walks [RankSliceLoIndex, oosCutoff) via a new m_dbBackfillStopIndex, clamped at both ends so a degenerate slice yields an empty walk rather than one that wanders into graded bars. Verified no reference to the old helper survives. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 22:58:55 -04:00
int m_dbBackfillStopIndex; // inclusive floor - the ranking slice's newest bar
int m_dbBackfillBars;
int m_dbBackfillFired; // rows written, for the completion log line
fix: the DB backfill could never run, and HEAD did not compile Four defects in 64c5dd5/1a05e63, found by review + a baseline compile. Goals 1-8 of that session are unchanged; this makes 6 and 8 actually reachable. 1. HEAD DID NOT COMPILE - 6 errors. CControlPanel::Minimize/Maximize were declared `virtual bool ... override`, but CAppDialog declares both as `virtual void` (Controls\Dialog.mqh). errors 265 + 404 on each, plus 151 on `bool ok = CAppDialog::Minimize()`. Return type is void now; there was never a success flag to forward. Verified: 0 errors, 0 warnings. 2. THE BACKFILL COULD NEVER ADVANCE, and neither could the OOS continual simulation (that one has been dead since it was written). Both are armed at the instant convergence is declared, and both advance only from inside Train(), one chunk per call. But ScheduleTrainingIfNeeded's only per-tick ArmStudyEvent site sits in the `else` of a branch taken whenever m_trainingComplete is set and m_trainRunActive is clear - which is exactly the state FinalizeTrainRun() leaves behind one line before they are armed. Train() was never called again, so the walks sat at their start index forever: no "simulation complete" line, and not one row written to the DB this feature exists to fill. Only a manual Resume/Retrain unstuck them. Both flags now keep the model schedulable. 3. IN AI_HYBRID - the mode this ships in - the backfill was never even armed. Ensemble members deploy at Train() ENTRY and return immediately (so no era is wasted), which skips the era-end block the backfill was started from. All four members were a no-op for a second, independent reason. Armed on the ensemble deploy path too, from m_resumeBars/m_resumeOosCutoff. 4. RE-RUNS DUPLICATED ROWS. RegisterSignal inserts unconditionally - no key, no duplicate check - and m_dbBackfillDone is in-memory, so every later attach that retrained to convergence wrote a second full set of rows for the same bars. The ranking would count one bar once per model that ever deployed, weighting superseded opinions as heavily as the live one. A .dbfill marker stamps the deployed era; written only on completion (an interrupted walk redoes itself rather than ranking a partial window) and deleted with the other sidecars on reset-weights. Also: WarmBlocking's timeout was silent, which restored the exact silent pin failure it was added to prevent - it now says so in the journal, and returns true for "no reference pairs to wait for" so the warning stays rare enough to be read. Not addressed, needs a decision: the backfill scores the OOS window with the checkpoint that was SELECTED as best on that same window, then writes those win rates into the table filter weights rank on - the selection set consumed twice, undiscounted, while the deploy gate right next to it applies a family-wise correction for exactly that effect. The rows are also simulated triple-barrier outcomes at today's spread sharing a table with realised fills. The completion log line now states both plainly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 21:25:51 -04:00
long m_dbBackfillEra; // era stamped into the .dbfill marker on completion
fix: drop the ranking slice for the calibration band; un-collapse the tiers NOT COMPILED - user compiles. (1) THE RANKING SLICE IS GONE. It reserved 20% of the OOS window so the pattern-DB backfill would read bars the deployed checkpoint was not SELECTED on. That objection stands; carving a new region to answer it did not. The calibration band already has every property the slice was buying: never trained on | never graded by pass 3 (which walks [0, oosCutoff) and so never reaches it) | never seen by the deploy gate | purged by a full label horizon on BOTH sides | and larger besides - 1,684 bars vs the ~970 carved So the backfill now walks [calibLo, calibHi) and pass 3 goes back to grading the entire OOS window, exactly as before any of this. The gate gets its full sample back (~10% of a sigma), the split loses a region, and the failure mode found an hour ago - a reserved region silently blanking ~10 months of chart arrows, because arrows are only drawn on bars pass 3 grades - becomes impossible. One impurity, stated in the completion log rather than hidden: m_dirConfThreshold is FITTED on that band and the walk applies it to decide which bars fired, so coverage there is mildly optimistic. One scalar under a coverage floor, against checkpoint selection over hundreds of eras. This backfill IS the deploy-time warm-up: it runs right after FinalizeTrainRun() restores the deployed weights, so it scores with exactly what is about to trade. (2) EVERY CALL WAS TIER 0, AND IT WAS ARITHMETIC. ConfidenceTier() quartiles [floorConf, 1] where floorConf = 1/3 - the lowest magnitude a 3-way softmax winner can hold. But it was fed CalibratedConfidenceMagnitude(), which multiplies by m_confidenceCalScale, clamped to [0.3, 1.5]. That lower clamp is BELOW 1/3. Whenever calibration bottoms out, t goes negative and MathMax(0, ...) pins every call to tier 0. Which is what the live run does. m_confidenceCalScale is EMA'd toward empiricalAccuracy / avgClaimedConfidence; with the model over-calling Neutral, 3-class agreement sits near 10% against a claimed confidence near 0.9, so the ratio is ~0.11 and clamps to 0.3 every era. Logged: tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0) 828 calls, one bucket - the four tier weights and the entire per-tier pattern-DB ranking reduced to a single number. The backfill was feeding a mechanism that structurally could not rank. Tiering now reads the RAW head magnitude, which genuinely lives on the [1/3, 1] range these bounds were written for. Calibration keeps its real jobs - AIConfidence() for MM sizing and SignedAIConfidence() for the vote are unchanged. STILL OPEN, deliberately not touched here: the calibration TARGET itself. empiricalAccuracy is 3-class agreement, which is the wrong quantity to scale a DIRECTIONAL confidence against - it counts a Neutral class that is 0.19% of labels. The honest target is the win rate on the calls the confidence describes (directional precision), with the claimed-confidence average taken over those same called bars. That needs a new accumulator and it interacts with the Neutral over-calling being fixed elsewhere, so it wants one clean run first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 23:06:11 -04:00
void StartPatternDatabaseBackfill(int bars, int totalIter, int oosCutoff);
void AdvancePatternDatabaseBackfill(void);
//--- same resumability problem one level up: TuneIndicatorsAndTrain()'s own trial loop calls
//--- Train() per trial and used to assume each call ran an entire trial to completion synchronously
int m_tuneTrialIndex; // -1 = no multi-trial tuning run in progress
double m_tuneBestOosForecast;
bool m_tuneLastTrialWasWin;
bool m_tuneHaveBestCheckpoint;
datetime m_tuneStartTrainBar;
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//=== Filter-based indicator auto-tuner (see TuneIndicatorsByFilter) =============================
//--- Replaced a genetic + successive-halving search on 2026-08-01. See TuneIndicatorsByFilter()
//--- for the measurements and the honest limit.
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
bool m_tuneFilterDone; // the one-shot filter pass has run for this model
//--- Mutual information between one feature column and the 3-class label, and the whole-vector score.
double FeatureColumnMI(const double &vals[], const int &labels[], int n);
//--- Returns the MEAN per-feature marginal MI. "0.001 nats" means nothing on its own; "0.1% of
//--- the label's entropy" is a magnitude anyone can act on.
fix(autotune): MI scorer read an array nobody filled; add the permutation floor THE TUNER WAS A SILENT NO-OP. Every chart logged auto-tune complete - 17 candidate settings scored in ~139s, feature/label mutual information 0.0000 -> 0.0000 nats (no improvement) 0.0000 is not a weak result, it is a broken measurement: finite-sample MI is biased UPWARD, so even pure noise scores above zero. Cause: ScoreCurrentParamsByMI called BufferTempDataCompute(), which APPENDS the bar's features to TempData and never touches m_featureCache - only the caching wrapper BufferTempData() writes that array. It then read m_featureCache, which ReInitADIndicators had just invalidated. Every column came back constant, FeatureColumnMI returned 0 for all of them, and all 17 candidates tied at exactly zero. 139 s per chart to return the settings it started with. Now reads the values back out of TempData, where they actually land. And an exactly-zero best score is called out as a fault rather than reported as "no improvement", because that is what it is. ADDED: a PERMUTATION BASELINE, which is the diagnostic this project has been missing. MI's finite-sample bias is ~(bins-1)(classes-1)/(2n) nats - at these sample sizes the same order as any real edge in this domain - so a raw MI figure is uninterpretable on its own. Shuffling the labels destroys every genuine association while leaving sample size, binning and class proportions intact, so the score it produces IS this dataset's noise floor, measured rather than approximated. The log now reads feature/label information - X nats against a shuffled-label floor of Y and says outright whether the features carry usable information about the target. It needs no training, no topology and no convergence, so unlike every accuracy number in this codebase it cannot be confounded by an optimizer or an objective. If the score sits on the floor, no change of architecture can help - which is the question the last three days of zero-edge results have been circling. DEPLOY FLOOR: `dirPrecPct > chancePrecPct` passed anything above chance by any amount. At ~11,000 directional calls the standard error of the precision estimate is ~0.4pp, so that gate was accepting sub-one-sigma noise - the perceptron deployed at edge +0pp on 2026-08-01. Now requires EDGE_MIN_SIGMAS (2.0) standard errors above chance, computed from the actual call count, so the bar scales with the evidence instead of needing a hand-picked constant. Recorded with it, because it is why chance is the right reference at all: under a driftless random walk P(touch +k*ATR before -m*ATR) = m/(m+k), and the break-even win rate for a k:m reward:risk trade is ALSO m/(m+k). The label's own base rate IS the break-even rate, at every SL/TP setting. So "beats chance" and "is profitable" are the same test, and no choice of SL/TP can manufacture an edge - only prediction can. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:05:50 -04:00
double ScoreCurrentParamsByMI(bool shuffleLabels = false);
//--- The same work split in two, so the permutation test can extract the sample ONCE and reuse
//--- it for every null draw. The sampled range is trimmed by MiShiftPad() at both ends - a FIXED
//--- amount, never by |offset| - so every build enumerates the same bars in the same order and
//--- two builds can be compared row by row.
diag: MI feature-lag profile - close the blind spot in every MI verdict so far BuildMiSample samples features from ONE bar. So every "MI is at the noise floor" result this codebase has produced - including yesterday's p=0.18 on SP500 H1 - described the ENTRY BAR's 31 features only, while the network is fed 20 bars of them. If information lived at lag 7 and not lag 0, the report would have said "no signal" while the model could still learn. The diagnostic we have been making decisions on had a blind spot exactly the width of the input vector. Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as the existing labelBarOffset and is not interchangeable with it. Shifting the LABEL changes which trade is predicted, so at any non-zero offset the features sit inside the labelled window and the score is lookahead - that is precisely what the alignment scan measures and correctly reports (4.7x more knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label pinned to the entry bar, so every row stays causal. ReportFeatureLagProfile() then scores k = 0..historyBars against the same block-permutation null and reports the deepest lag that clears it - the lookback the data supports, versus the 20 that was picked by hand and never measured. The null is redrawn PER LAG: finite-sample MI bias moves with the realised class counts and bin occupancy, and different rows survive the validity checks at each lag, so one shared floor would be right for lag 0 and wrong everywhere else. Draw count is reduced accordingly (40, not 200) since cost is draws x historyBars; this figure decides a lookback, never a trade. MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that makes two builds comparable row by row. Read-only - no input, topology or label change, so no retrain. Both builds 0/0. Build tag lag-profile-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 13:51:39 -04:00
int BuildMiSample(double &cols[], int &labels[], int labelBarOffset = 0,
diag: is "optimal SL/TP" learnable? Score the features against excursions Proposed direction: train the net to predict entry/SL/TP that maximise return and minimise drawdown, rather than to classify direction. Before rebuilding a head, measure whether the target is learnable at all. That question splits into two that behave nothing alike: HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility clustering is about the most robust regularity in markets. WHICH WAY it goes first (the asymmetry) - direction, which is what every noise-floor verdict in this project has been about. Expectancy comes ONLY from the second. The first buys position sizing and drawdown control - worth having under prop-firm limits, but not an edge: exit management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with expectancy FLAT. Crucially this is NOT already answered. Every MI figure here scored the triple-barrier label, i.e. one specific question at one fixed geometry. A noise-floor result there says nothing about whether excursion MAGNITUDE is learnable - different target, different answer. Four targets, and the verdict is the CONTRAST, printed explicitly because the dangerous misreading of "UP clears" is "we can predict profitable trades": RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that SHOULD clear. Every prior verdict here lacked a control expected to pass; a range target at the floor indicts the measurement, not the market. UP / DOWN - MFE / MAE. ASYMMETRY - up-dn, the only one that can pay. Collected inside the walk the label already does (one max, one min per bar). The early-out when both barriers resolved is GONE: it would have truncated the excursions at whichever bar tripped the last barrier, making the measurement a function of the CURRENT SL/TP - the circularity this is trying to escape. The loop was already bounded by the horizon, so only the average cost moves. Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is fat-tailed and fixed-width bins would put nearly every row in bin 0; it also pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and to the barrier label's ~1.02 instead of confounded by class balance. Two bugs fixed in this code before it ever ran, both of which would have produced a plausible quiet wrong answer rather than an error: - TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the accumulators were reset, so one bar's excursions would be cached under another bar's index. Cleared at the top now, ahead of every return. - An unresolvable bar is still flagged as labelled but carries excursions of exactly 0. Under equal-frequency binning a block of identical zeros drags the lowest cut onto zero and a third of the sample lands in one uninformative bin - a depressed score that reads as "not predictable", a false negative in the direction that would wrongly kill the idea. Rows where both excursions are zero are dropped; price cannot travel zero both ways over a whole horizon. Read-only diagnostic. No topology or label change: no retrain of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 10:22:41 -04:00
int featureBarOffset = 0, int target = MI_TARGET_BARRIER);
//--- Is an "optimal SL/TP" head learnable? Scores the features against excursion magnitude and
//--- asymmetry instead of the barrier class - a different question, see the definition.
void ReportExcursionInformation(void);
feat: derive the ATR multiples from measured excursions - no hardcoded geometry The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in 3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of eleven guesses is not deriving anything. WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It ranks pairings by how predictable their OUTCOME is - a question about direction. The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x. Hence the scan failing its own gate on every run, and its "winner" wandering 2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is strongly measurable, so derive the geometry from that instead. stop = q25 of measured ADVERSE travel (ordinary noise does not reach it) target = q50 of measured FAVOURABLE travel (reached ~half the time, by construction, inside the horizon) Continuous, in ATR units, superseding the enum multiples. Reachability ("target on X% of bars, stop on Y%") and the implied break-even are printed so the choice is auditable rather than trusted. FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the horizon with the target (first-passage time grows with the band) and the excursions are measured OVER the horizon, so target -> horizon -> excursions -> target is a real loop - deriving once sizes the target from travel measured under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at 3 passes, and says so if it does not settle. Does NOT create expectancy, and the log says as much: chance precision equals break-even at every geometry (m/(m+k) on both sides). It buys a target the market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio forces a target the market rarely reaches, it WARNS rather than overriding - the ratio is the user's risk policy, so the honest move is to state its cost. That is the collision that once rejected 100% of setups. Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg files written earlier today still load (their length guard finds no doubles) and a model that carries them was trained on them and never re-derives. Also fixes a message from e5ceed6 that claimed "this model resumed from disk" unconditionally - it printed above a "seeding era 0" line on a brand-new model, because the branch fires whenever the cache is not built, which is equally true before a fresh model's first prebuild. A diagnostic that misreports its own trigger is worse than one that says nothing: it gets quoted back as evidence. FORCES A FULL RETRAIN (labels change). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 12:06:25 -04:00
//--- Sets the ATR multiples from the measured MFE/MAE quantiles instead of the mode enums. Returns
//--- false (and leaves the configured pair standing) when there are too few resolved excursions.
bool DeriveBarrierGeometry(void);
//--- LAG PROFILE: how far back the features still say anything about the entry they precede. WHY
//--- THIS WAS MISSING AND WHY IT MATTERS: BuildMiSample samples ONE bar.
diag: MI feature-lag profile - close the blind spot in every MI verdict so far BuildMiSample samples features from ONE bar. So every "MI is at the noise floor" result this codebase has produced - including yesterday's p=0.18 on SP500 H1 - described the ENTRY BAR's 31 features only, while the network is fed 20 bars of them. If information lived at lag 7 and not lag 0, the report would have said "no signal" while the model could still learn. The diagnostic we have been making decisions on had a blind spot exactly the width of the input vector. Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as the existing labelBarOffset and is not interchangeable with it. Shifting the LABEL changes which trade is predicted, so at any non-zero offset the features sit inside the labelled window and the score is lookahead - that is precisely what the alignment scan measures and correctly reports (4.7x more knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label pinned to the entry bar, so every row stays causal. ReportFeatureLagProfile() then scores k = 0..historyBars against the same block-permutation null and reports the deepest lag that clears it - the lookback the data supports, versus the 20 that was picked by hand and never measured. The null is redrawn PER LAG: finite-sample MI bias moves with the realised class counts and bin occupancy, and different rows survive the validity checks at each lag, so one shared floor would be right for lag 0 and wrong everywhere else. Draw count is reduced accordingly (40, not 200) since cost is draws x historyBars; this figure decides a lookback, never a trade. MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that makes two builds comparable row by row. Read-only - no input, topology or label change, so no retrain. Both builds 0/0. Build tag lag-profile-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 13:51:39 -04:00
int ReportFeatureLagProfile(void);
//--- Bars trimmed from each end of every MI sample. Must cover the largest offset any caller
//--- asks for: the alignment scan's MI_ALIGN_MAX_SHIFT and the positive control's horizon/4.
int MiShiftPad(void) const
{
diag: MI feature-lag profile - close the blind spot in every MI verdict so far BuildMiSample samples features from ONE bar. So every "MI is at the noise floor" result this codebase has produced - including yesterday's p=0.18 on SP500 H1 - described the ENTRY BAR's 31 features only, while the network is fed 20 bars of them. If information lived at lag 7 and not lag 0, the report would have said "no signal" while the model could still learn. The diagnostic we have been making decisions on had a blind spot exactly the width of the input vector. Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as the existing labelBarOffset and is not interchangeable with it. Shifting the LABEL changes which trade is predicted, so at any non-zero offset the features sit inside the labelled window and the score is lookahead - that is precisely what the alignment scan measures and correctly reports (4.7x more knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label pinned to the entry bar, so every row stays causal. ReportFeatureLagProfile() then scores k = 0..historyBars against the same block-permutation null and reports the deepest lag that clears it - the lookback the data supports, versus the 20 that was picked by hand and never measured. The null is redrawn PER LAG: finite-sample MI bias moves with the realised class counts and bin occupancy, and different rows survive the validity checks at each lag, so one shared floor would be right for lag 0 and wrong everywhere else. Draw count is reduced accordingly (40, not 200) since cost is draws x historyBars; this figure decides a lookback, never a trade. MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that makes two builds comparable row by row. Read-only - no input, topology or label change, so no retrain. Both builds 0/0. Build tag lag-profile-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 13:51:39 -04:00
//--- Also covers m_historyBars, because the lag profile shifts the FEATURES that far back and every
//--- build must still enumerate the identical bar set (see BuildMiSample's fixed-pad note - padding
//--- by the requested offset instead is what voided the positive control on 2026-08-02).
return MathMax((int)MathMax(m_historyBars, 0),
MathMax(MI_ALIGN_MAX_SHIFT, MathMax(m_barrierHorizonBars, 1) / 4));
}
diag(autotune): five permutations was still a coin flip - use a real test The 5-draw z-score shipped an hour ago disproved itself on its first run. All four charts scored the IDENTICAL 0.00401 nats on identical features and identical labels - and reported z of +1.3, +2.0, +4.0 and +4.7. Two "AT THE NOISE FLOOR", two "a real association", same data. The entire swing came from estimating the null's spread from five draws, where the standard deviation of the standard-deviation estimate is ~35%: the denominator was noisier than the effect it was judging. Replaced with an empirical permutation test. 200 draws, p counted by rank with the +1/(B+1) correction (Phipson & Smyth 2010) so p is never reported as exactly zero - no normality assumption and no spread to estimate. The strongest single column is tested against the null distribution OF THE MAXIMUM, which corrects for scoring 26 features at once by construction and is far less conservative than Bonferroni. Affordable because BuildMiSample is now split out of ScoreCurrentParamsByMI and runs ONCE for the whole test - every draw reuses that sample and costs a relabel plus 26 histogram passes, not 2000 feature extractions. The coordinate sweep still calls the combined form, which is correct there: each candidate changes the indicator settings, so its features really do have to be re-extracted. The verdict line keeps both questions apart and prints both answers: the p-value for "is it real", the excess as a percentage of H(Y) for "is it big enough to trade". At n=2000 those can disagree, and collapsing them into one word is how a worthless effect gets called a discovery. Compiles 0 errors / 0 warnings. Build tag permtest-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:45:46 -04:00
double ScoreMiSample(const double &cols[], int &labels[], int n, bool shuffleLabels);
//--- The permutation test + verdict, split out of the tuner so it is NOT gated on era 0 with it - see
//--- the definition. Read-only; runs once per attach, whether or not the sweep did.
void ReportFeatureLabelInformation(void);
//--- Smallest BarsCalculated() across the ENABLED tunable indicators, or -1 when none is on. The
//--- tuner reports this so "the parameter change did not reach the features" can be told apart
//--- from "it reached them but they weren't ready".
fix: make the indicator tuner actually measure, and gate what it installs ROOT CAUSE of the zero spread measured on SP500 H1 2026-08-07 (all 17 candidates returned exactly 0.00359 nats): the tune loop re-inits the indicators and then scores, with no RefreshData() between. ReInitADIndicators() does its part - Create() builds a NEW handle carrying the new parameters, and the feature cache is flagged stale so features really are recomputed. But BufferTempDataCompute() reads the CIndicatorBuffer objects, and only Refresh() copies data out of a handle into those. So every candidate was scored on values still held from the PREVIOUS handle. My earlier guess in the diagnostic ("suspect the feature cache") was wrong: the cache invalidation works. Two things land together, because neither is safe alone: 1. RefreshData() after the re-init, so a candidate is scored on its own features. 2. A SELECTION GATE on the install. bestScore is a MAXIMUM over candidates, and the maximum of N draws from a null beats its incumbent almost every time - so "it beat the incumbent" installs noise. This selector is the highest-stakes of the three found in this audit because it ACTS: it overwrites the user's configured indicator settings and forces BuildFreshTopology(), so the network then trains on whatever the noise picked. Fixing (1) without (2) would have made a dormant bug actively harmful. The gate draws the winner's own permutation null once, then corrects the p-value for having chosen it out of N with Sidak: p_family = 1 - (1-p)^N. Sidak rather than the max-of-N resample used by the geometry scan because each candidate here has a DIFFERENT feature set, so their draws cannot be pooled; Sidak needs only the one null. Exact under independence, mildly anti-conservative under positive dependence - stated in the comment rather than hidden. A rejected winner restores the configured settings, which best[] cannot do since the descent mutates it. Also reports the least-ready tunable handle's BarsCalculated(). IndicatorCreate() calculates asynchronously, so if the spread is STILL zero the handles simply are not done and the tuner needs to yield between candidates rather than score them back to back - a state machine like the label prebuild. That distinction is now readable from the log instead of requiring another guess. No input, topology or label change: no retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 09:31:06 -04:00
int TunableBarsCalculated(void);
//--- Same number, plus HOW MANY tunable indicators were actually consulted.
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
int TunableBarsCalculated(int &enabled);
fix(indicators+panel): the dead handle is MEASURED now - recreate it; and order the ensemble panel by member, not by who published first THE ANSWER, off the instrumentation added in be39674, first run: ConvLSTM [HYB-2484]: TUNABLE INDICATOR REPORTS NO CALCULATED BARS - 1 tunable indicator(s) enabled and the least-ready answers BarsCalculated()=-1 ... Per-indicator depth: price=33982 MA=-1 ZigZag=33982 ATR=33982 MA=-1 with price, ZigZag and ATR all at full depth. **The handle is INVALID, not short.** Same line on USDJPY (price=50179 MA=-1). Depth was never the problem; the previous session's five theories were all answering the wrong question. And it is per-member, not per-chart: LSTM-2484 ran the 34-candidate auto-tune on that same chart at 15:24:18 and went on to train normally (feature health, 51 features, excursion head) reading the same indicator. Only ConvLSTM's handle - the last member constructed - was dead. WHY is still not established. All four members request ADMovingAverage with identical params, so MT5 hands them the SAME refcounted handle, and the tuner's inner loop is Create-then-IndicatorRelease over exactly that shared handle; that is the obvious suspect and it is NOT yet proven, so this commit does not act on it. 1. IndicatorDepthReport() NOW PRINTS HANDLE NUMBERS, not just depths. "MA=-1" says the handle is dead. "MA=-1(h12)" against another member's "MA=33982 (h12)" says it is the SAME handle and someone released it; "(h-1)" says it was never created. That is the difference between a refcount bug and a creation failure and it is one field. This is the measurement the shared-handle suspicion needs before anyone acts on it. 2. RECREATE A DEAD HANDLE INSTEAD OF SWEEPING AGAINST IT. A member that cannot read its own indicator must rebuild it. RepairDeadIndicatorHandles() re-Creates only the ENABLED tunables reporting BarsCalculated() < 0 - a merely COLD indicator (valid handle, 0 bars) is left alone to warm up the normal way. It does NOT release first: -1 means the terminal no longer knows the handle, so there is nothing to give back, and MT5 recycles handle VALUES so releasing a stale one could decrement whatever now owns that number. 30s cooldown, because every ServableBars() consumer reaches it including live inference on every tick. The feature cache is dropped with it, and the log names before/after depths. Cause-agnostic on purpose. Whatever is killing the handle, sweeping 50,163 bars against a buffer that answers EMPTY_VALUE at every index - then discarding the era and doing it again - is not a recovery. 3. THE SWEEP NOW HOLDS ON A DEAD HANDLE. ServableBars() keeps answering `want` (its contract; live inference and online learning have their own refusal paths and a 0 there reads as "no history at all"). SettledBars() - the training sweep's entry, the one caller that can afford to wait - returns 0 instead, so Train() holds and reports rather than burning a full-history pass it is guaranteed to throw away. A recreated handle is cold, so it primes through the existing settle path on the next call. If the repair fails the member holds indefinitely and says so every minute, and be39674's barrier liveness escape releases the rest of the ensemble after 12 minutes - which is the correct degradation and is exactly what the log shows happening. 4. THE PANEL ROWS WERE ORDERED BY WHO PUBLISHED FIRST. Reported on XAUUSD: LSTM, ConvLSTM, Perceptron, Convolutional instead of Perceptron, Convolutional, LSTM, ConvLSTM. ClaimEnsemblePanelSlot() handed out the next free row on each member's FIRST PublishStatus() call, so the order was a race - the members busy sweeping published before the ones sitting idle at the era barrier, and be39674 sharpened it by (correctly) making a held member stop writing the terse line. Rows are now keyed to m_ensembleIndex, the registration/construction order, which is fixed for the life of the chart. Claimed on every publish rather than once, so it is idempotent and refreshes the tag for a member whose ID was not final when it first published (the config-tag suffix is appended during InitIndicators, after EnsembleMember() registers). Unclaimed rows are skipped by the render and excluded from the model count, so a member that has not published yet leaves no gap and shifts nobody. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:40:00 -04:00
//--- Re-Create any ENABLED tunable indicator whose handle the terminal no longer recognises
//--- (BarsCalculated() < 0). Returns true when something was actually rebuilt.
bool RepairDeadIndicatorHandles(void);
//--- `want`, clamped to what the indicators can actually serve. THE single gate in front of
//--- every ResizeBuffers() call site (train, live inference, chart rescan, research export).
fix(depth): route EVERY ResizeBuffers call site through one indicator-depth gate 1dda479 clamped the training sweep. It left five other paths asking the indicators for a depth they cannot serve, and on a live account the quiet ones are worse than the stall was - a stalled chart is visible, a chart trading on a degraded feature window is not. ServableBars(want, context) is now the single gate, and all six go through it: training sweep clamp, floored at TRAIN_MIN_CLAMPED_BARS (below that a small positive BarsCalculated is warm-up, which m_coldSweepTick owns) label prebuild clamp - labels come from price/ADZigZag and would survive a capped MA, but ResizeBuffers sizes EVERY buffer and a failed CopyBuffer leaves m_MA EMPTY for the next reader, so this path could silently re-break the block Train()'s clamp just fixed live inference HOLD. Below `need` the swing block takes its degraded path and inference runs on a different feature distribution than the model was fitted on. This EA sizes real positions off that output, so no signal beats a mismatched one online learning HOLD, same reason and worse - this path WRITES to a live trading model, so a mismatched (features, label) pair is not a wrong arrow, it is a wrong weight update that compounds every bar chart rescan clamp - SIGNAL_RESCAN_LOOKBACK_BARS is 5000 and MT5's smallest "Max bars in chart" is also 5000, so this one is genuinely reachable; uncapped it repaints the window all-Neutral research export clamp before the emptiness test, so a capped symbol exports the depth it has rather than writing a CSV with a dead feature block - an artefact that looks complete and is silently wrong Both HOLDs are insurance, not expected states: `need` tops out near 1,152 bars (16 + 750 + 384 + 2) against a 5,000 floor on the terminal setting. They exist so the failure mode is unreachable rather than merely unlikely. Not changed: a genuinely SHORT price history still takes the old degraded path at every site. That is pre-existing behaviour and narrowing it would mute charts that trade today, so it stays a separate decision rather than a side effect of this fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:33:16 -04:00
int ServableBars(int want, string context);
//--- ServableBars() with a WAIT in front of it, for the paths that can afford one (the training
//--- sweep and the label prebuild). Returns >0 = the depth to use, or 0 = "not settled, come
//--- back later".
feat(depth): prime -> settle -> sweep, and name which handle is short "Max bars in chart" is set to Unlimited, so the static-terminal-limit reading in 1dda479 was wrong. Two other candidate causes are falsified too: the price series is fully downloaded and flat (USDJPY 50,162 -> 50,163 over 80 minutes, i.e. one new H4 bar), and the handles have been stable since 13:07 with zero windows for the 17 minutes after, so it is not download-in-progress and not handle churn. The MA period tops out at 200 (ADIndicatorTuner MA_PERIOD_PRESETS) against ~50k bars, so it is not indicator cost either. What IS verified stays verified: CopyBuffer past the calculated depth fails outright rather than short-reading, so the buffer holds nothing and every index reads EMPTY_VALUE; m_MA is the only CiCustom whose block REJECTS on that (m_ADZigZag neutral-fills, RSI/MACD/Ichimoku/ATR are built-ins); the wall is therefore feature 25 of every bar, exactly as the "24 of 832" stall lines said. And it is depth-correlated: 16k-bar charts train, 34k/50k get zero windows forever. So the WHY is still open, and this fix does not depend on it. Per the user's protocol: the request itself is the primer, so prime at full depth, then poll TunableBarsCalculated() every 3s and hold the sweep until it stops changing (3 steady probes), then use whatever it settled at. Bounded at 10 min, and a give-up is logged as a give-up so an abandoned depth is never mistaken for a settled one. This supersedes 1dda479's clamp on the two training paths, which snapshotted a value that may still have been climbing; the clamp remains for the paths that cannot wait (inference/online/rescan/export, see 7e63a8b). The load-bearing part is what does NOT happen while waiting: no sweep. A 50k-bar feature scan starves the indicator threads the request just woke, which is how the failure sustained itself for 40 minutes at a time - discard era, re-sweep, discard, which is the 0->100% oscillation on the panel. Also adds IndicatorDepthReport(): per-handle BarsCalculated() on the priming, cap and stall lines. The logs proved WHICH FEATURE died but never WHICH HANDLE was short, so the cause had to be inferred - and was guessed wrong twice. The next occurrence reads it off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:44:27 -04:00
int SettledBars(int want, string context);
//--- Per-indicator BarsCalculated(), for the cap/priming/stall lines. It answers that directly.
//--- Which BLOCK a feature slot belongs to, e.g. "spread[1]".
fix(telemetry): FEATURE HEALTH said "f30", which is a puzzle rather than an answer The report has flagged f30 as mostly-zero (78%) on every member of every run for days. Establishing what f30 actually IS took reconstructing the emission order across three files, and I got it wrong on the first attempt - guessed RSI, then MACD, both wrong because those feature blocks ship disabled. It is spread[1], the spread CHANGE ratio, and 78% exact zeros is exactly what that should read: the broker quotes the same spread on consecutive bars most of the time, so the change is exactly 0. Benign, and it cost two wrong answers to say so. The report now names the block - "spread[1] (78%)" instead of "f30 (78%)". The walk lists every block in the order BufferTempDataCompute emits them with the widths Topology.mqh's m_neuronsCount sum declares, which makes this a third place that has to stay in step with those two. So it does not stay in step silently: the widths must total m_neuronsCount, and when they do not the layout has drifted and every name past the drift point is wrong - so it returns "f<slot>?" and names nothing rather than naming confidently and incorrectly. A wrong name is worse than a bare index. This is the same lesson as cb30360 (print the resource's IDENTITY, not just its state), applied to the feature vector. The alt-block hint in the header goes away with it - it existed to disambiguate one block, and every block is disambiguated now. Reporting only, no behaviour change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 14:42:02 -04:00
string FeatureSlotName(const int slot);
feat(depth): prime -> settle -> sweep, and name which handle is short "Max bars in chart" is set to Unlimited, so the static-terminal-limit reading in 1dda479 was wrong. Two other candidate causes are falsified too: the price series is fully downloaded and flat (USDJPY 50,162 -> 50,163 over 80 minutes, i.e. one new H4 bar), and the handles have been stable since 13:07 with zero windows for the 17 minutes after, so it is not download-in-progress and not handle churn. The MA period tops out at 200 (ADIndicatorTuner MA_PERIOD_PRESETS) against ~50k bars, so it is not indicator cost either. What IS verified stays verified: CopyBuffer past the calculated depth fails outright rather than short-reading, so the buffer holds nothing and every index reads EMPTY_VALUE; m_MA is the only CiCustom whose block REJECTS on that (m_ADZigZag neutral-fills, RSI/MACD/Ichimoku/ATR are built-ins); the wall is therefore feature 25 of every bar, exactly as the "24 of 832" stall lines said. And it is depth-correlated: 16k-bar charts train, 34k/50k get zero windows forever. So the WHY is still open, and this fix does not depend on it. Per the user's protocol: the request itself is the primer, so prime at full depth, then poll TunableBarsCalculated() every 3s and hold the sweep until it stops changing (3 steady probes), then use whatever it settled at. Bounded at 10 min, and a give-up is logged as a give-up so an abandoned depth is never mistaken for a settled one. This supersedes 1dda479's clamp on the two training paths, which snapshotted a value that may still have been climbing; the clamp remains for the paths that cannot wait (inference/online/rescan/export, see 7e63a8b). The load-bearing part is what does NOT happen while waiting: no sweep. A 50k-bar feature scan starves the indicator threads the request just woke, which is how the failure sustained itself for 40 minutes at a time - discard era, re-sweep, discard, which is the 0->100% oscillation on the panel. Also adds IndicatorDepthReport(): per-handle BarsCalculated() on the priming, cap and stall lines. The logs proved WHICH FEATURE died but never WHICH HANDLE was short, so the cause had to be inferred - and was guessed wrong twice. The next occurrence reads it off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:44:27 -04:00
string IndicatorDepthReport(void);
fix(indicators): a dead handle and a priming one both read -1, so the repair report proved nothing The detector claimed "-1 means an INVALID HANDLE, 0 means created-but-never- calculated". This run disproved it with our own instrumentation: the repair line prints only when Create() RETURNED TRUE, and the depth it read microseconds later was BEFORE: MA=-1(h13) | AFTER: MA=-1(h13) A freshly created, valid handle read -1 - the value the model says is impossible for one. So BarsCalculated() < 0 does not mean "dead"; it also covers "valid, not calculated yet", and the trigger cannot separate them. Consequences, all fixed here: - The AFTER depth was re-read synchronously, when it can only be -1 or 0, so every repair looked like a failure and the line was unreadable either way. It now reports the handle NUMBER across the recreate instead. A changed number proves a new instance; SAME means MT5 handed back the same refcounted one, so it was never dead. - IndicatorDepthReport printed the handle number for MA alone. Every tunable gets one now, through a single IndicatorDepthField() - nine near-identical StringFormat calls collapse to one. - The comment justifying "never release before re-creating" rested on the claim just disproved. The decision stands, the reason is restated: given the ambiguity, releasing is the dangerous half - a recycled number would decrement whatever owns it now and CAUSE this outage - while re-creating a live handle only leaks a reference on a path that fires a few times a session. The before-handle is captured on its own line, never as a sibling argument to the Init* call: MQL5 does not define argument evaluation order. Behaviour is otherwise unchanged - same trigger, same cooldown, same recreate. Only what gets reported changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 13:38:02 -04:00
//--- One field of that report, and one line of the repair report. Both print the handle NUMBER:
//--- a depth of -1 cannot separate "freed under this member" from "created but not calculated yet",
//--- and the number can.
string IndicatorDepthField(const string name, const int depth, const int handle);
int NoteHandleMove(const string name, const int oldHandle, const int newHandle, string &moves);
bool m_miReportDone;
//--- Eras the MI report has waited for the cross-asset panel to exist, so it describes the SAME
//--- feature vector training uses. Bounded, so a terminal that never syncs the reference symbols
//--- still gets its diagnostics rather than silently getting none.
int m_miReportDeferrals;
feat(labels): measure which barrier is predictable at entry, don't guess The alignment scan settled the shape of the problem: 4.7x more is knowable 5 bars into a 128-bar window than at the entry the model actually trades. A 6xATR target reached over 128 bars is decided overwhelmingly by what happens DURING the window, so whatever the entry state knows is buried under 128 bars of later noise. That is a property of the TARGET, and it is why four different architectures all landed on precision exactly equal to the base rate - no topology can undo it. So measure the target. For each SL/TP pairing a user can actually select, relabel the same sampled bars and score how much the SAME features say about THAT outcome at entry. Seconds, no training, no topology, and it runs on the diagnostic path that already exists. Ranked on excess over its OWN null as a share of its OWN H(Y), never on raw nats: each geometry has a different class balance, hence a different finite-sample bias and a different amount of information there to find, so raw MI would rank the most BALANCED label rather than the most PREDICTABLE one. The break-even win rate m/(m+k) is printed beside each so the ranking is read next to the bar the model must clear. Stated in the output because it is the easy thing to get wrong: chance precision EQUALS break-even at every geometry, so a tighter target does not hand you expectancy. It buys predictability - less noise piled on top of what the entry state knows - which is the one thing changing topology cannot do. Read-only by construction: it relabels a sampled copy via TripleBarrierLabel(), never writes the label cache (which belongs to the configured geometry), and restores the horizon and overrides it borrowed. The overrides apply only when BOTH are positive, so a half-set pair can never silently relabel a live run. Compiles 0 errors / 0 warnings, standard and Market. Build tag geometry-scan-v1. Redeploy only - no retrain to READ the ranking. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:32:04 -04:00
//--- Ranks every selectable SL/TP pairing by how much the SAME features say about THAT barrier
//--- outcome at ENTRY time - see the definition. Read-only: it relabels a sampled copy, never the
//--- label cache, and restores the barrier state it borrowed.
void ReportBarrierGeometryScan(void);
//--- Scan overrides consulted by BarrierMultiples(). Both > 0 or neither applies; 0 = off. Live only
//--- for the duration of ReportBarrierGeometryScan, and nothing persisted is keyed on them.
double m_barrierScanSlMult;
double m_barrierScanTpMult;
//--- true => BuildMiSample computes each label with TripleBarrierLabel() instead of reading the cache,
//--- because a hypothetical geometry's labels are by definition not cached.
bool m_barrierScanLiveLabels;
fix(labels): the geometry scan rewarded the labels it should reject First run named 3:10 on all four charts, at 2.3x the configured 2:6. That answer was wrong and the fault was the ranking statistic. 3:10 wants a horizon of ~swingMedian*30 (~320 bars) and gets BARRIER_HORIZON_MAX. Clamped, most trades never resolve, the unresolved remainder all lands in Neutral, and H(Y) collapses. The old statistic divided the excess BY H(Y) - so a collapsing denominator made the most degenerate label look like the most predictable one. Every geometry from 2:6 upward was already showing the clamped h128, and the two widest scored highest, which is the fingerprint of the artefact rather than of signal. Two fixes: Rank on the raw excess in nats. Subtracting each geometry's OWN measured null already removes the class-balance bias, which is the only thing the normalisation was ever needed for. Disqualify clamped geometries outright rather than ranking them down. The deployed EA holds until SL or TP with no bar limit, so a truncated label trains the model on a question the strategy never asks. They are still printed, marked '!', so the disqualification is visible instead of a silent omission - and the scan now says so explicitly when nothing eligible is left, because "the limit is the feature set, not the target" is itself the finding in that case. The scan also reports each geometry's directional share and timeout share now. A label nobody can trade is not a candidate however well it scores, and that has to be visible in the same line as the score. Compiles 0 errors / 0 warnings. Build tag geometry-scan-v2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:46:25 -04:00
//--- timed-out labels seen during one geometry's live relabel - see the scan's dir/to columns.
int m_barrierScanTimeouts;
//--- set by ComputeBarrierHorizonBars: this geometry needs MORE time than BARRIER_HORIZON_MAX allows,
//--- so its label truncates a trade the EA would hold to SL/TP. Disqualifies it from the scan.
bool m_barrierHorizonClamped;
double m_miBestColumn;
double m_miLabelEntropy;
diag(autotune): a positive control, and a scan that separates "no signal" from "signal knocked out of step" Four architecturally different networks landed on the same precision - Buy 23-25% against a 25.4% base rate, Sell 19-22% against 22.0% - while making completely different calls (HYBRID votes Sell on 69% of bars, PAI on 41%). Precision equal to the base rate is what INDEPENDENCE looks like, and precision under independence is fixed by the label distribution, not by the architecture, so all four converging on it is arithmetic rather than coincidence. Accuracy meanwhile tracks coverage exactly as independence predicts (31.1/30.3/25.0 predicted vs 31.8/28.9/24.6 observed for PAI/CONV/HYB). But "no information in the data" and "information destroyed upstream of every topology" produce that identical picture, and the MI test alone cannot tell them apart either. Two additions: POSITIVE CONTROL. Three "measurements" in this codebase have turned out to be silent no-ops that produced plausible numbers - the MI scorer reading an array nobody filled, the eval-mode guard that switched off the imbalance correction, the alternation gate whose premise was never true. So the estimator now has to prove it responds to a signal known to be present before any floor reading is believed: the label of a neighbouring sample row, ~19 bars away and far inside the 128-bar barrier horizon, so the two outcome windows overlap heavily and MUST be associated. Same binning, same estimator. Near the floor => every MI figure is void. ALIGNMENT SCAN. Re-scores against the label taken from bar i+k for k in -5..+5. A peak at k != 0 is a feature/label misalignment - an off-by-one in the label index, a horizon applied to the wrong bar, a feature window that lags what it claims - which would destroy the information before any topology saw it and would look identical in every accuracy number this EA prints. A flat profile says the features simply do not carry this target. The sampled range is trimmed by |k| at both ends so a shift is measured rather than an edge effect, and both bars must carry a real label. Also: BuildMiSample publishes its stride instead of the report recomputing that arithmetic (it would drift), and the control sizes its buffers from its own sample count rather than the caller's. Compiles 0 errors / 0 warnings, standard and Market. Build tag mi-control-align-v1. Redeploy only - no retrain, no model deletion; the diagnostic runs on resumed models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:12:10 -04:00
//--- bars between two consecutive MI sample rows, set by BuildMiSample - see its note.
int m_miStrideBars;
fix(diag): the symbol sweep was measuring its own sampling, not the market Twelve cells came back with higher-timeframe "signal" 5-9x anything on H1, at p=0.005. It was an artifact, and the sweep's own columns gave it away: excess tracked the sampling STRIDE almost monotonically, and the three D1 cells - stride collapsed to 1-5 bars against a 128-bar horizon, i.e. ~99% window overlap - were the three highest. Three flaws, all the same family: comparing numbers without the spread that belongs to them. 1. THE NULL ASSUMED INDEPENDENCE THE LABELS DO NOT HAVE. Triple-barrier labels overlap; two rows less than one horizon apart share most of their outcome window. A free Fisher-Yates shuffle destroys that dependence along with the association, making the null far narrower than the truth and handing out significance that isn't there - Lopez de Prado ch. 4 arriving through the back door of the significance test. Now permutes contiguous BLOCKS of at least one horizon, so the null keeps the autocorrelation and the p-value means what it says. It degrades honestly: severe overlap leaves few blocks, the null widens, nothing reaches significance. The block count is now printed, because THAT - not the row count - is the sample size a p-value rests on, and a warning fires under 30 blocks so "not significant" is not misread as "no signal" when it means "not enough independent history to tell". 2. THE POSITIVE CONTROL'S STRENGTH DEPENDED ON THE DATASET. It paired each row's label with the NEXT SAMPLE ROW's, whose distance is the stride - so on M5, where stride ran 160-717 bars against a 128-bar horizon, it was pairing two windows that never overlap. All three M5 cells duly reported a FAILED estimator and voided their own results with nothing wrong. A control whose strength varies with the cell cannot certify the cell. Now pinned to a quarter of the horizon, where ~75% overlap is guaranteed by construction. 3. THE LOOKAHEAD VERDICT HAD NO MARGIN. It flagged 7 of 12 cells on gaps of 0.00008-0.00040 nats against a measured null sd of ~0.00030 - noise, every one. Now requires 3 sd, the same discipline the deploy floor applies to precision. Compiles 0 errors / 0 warnings, standard and Market. Build tag blockperm-v1. Supersedes every number from the sweep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 15:11:40 -04:00
//--- independent label blocks the permutation null was built from (rows within one barrier horizon
//--- move together, so THIS - not the row count - is the sample size the p-value really rests on).
int m_miNullBlocks;
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
void TuneIndicatorsByFilter(void);
//================================================================================================
void FinalizeTrainRun(void);
//--- The "this is now THE model" persistence sequence, shared by every deploy path so they can't
//--- drift apart: weights (carrying the current m_trainingComplete flag), the pure-MQL5
//--- inference self-check, the calibration sidecar, and the EMA shadow.
void PersistDeployedModel(void);
//--- On hitting the per-run era cap: asks the operator whether to keep training (true) or deploy
//--- the best checkpoint and stop (false). Headless (tester/optimizer) can't show a dialog, so it
//--- returns false. See m_maxErasPerRun's declaration comment.
bool PromptContinuePastEraCap(double bestOos);
//--- variables
//--- training control, driven by the control panel (Warrior_EA.mq5); Train()/OnTickHandler
//--- poll these rather than being torn down/rebuilt, so pausing/stopping never loses in-memory state
bool m_trainingPaused; // true: Train() blocks between eras until unpaused
bool m_trainingStopRequested; // true: OnTickHandler stops scheduling new training passes
//--- true for ANY Strategy Tester run - a single backtest AND every optimization pass
//--- (MQL_TESTER): the run must NEVER train. Training + online continual learning happen only on
//--- a live chart, where this is false.
bool m_inferenceOnly;
//--- true only when the current Net weights came from a saved .nnw on disk, not from a freshly-
//--- built random topology.
bool m_modelLoadedFromDisk;
//--- Set by EnforceTopologyContract() when a just-loaded .nnw was built by a superseded
//--- architecture that cannot be repaired in place (currently: a different conv receptive field,
//--- whose weight tensor is a different SHAPE).
feat(ai): real conv receptive field + the reference's channel pool CONV's convolution used window = step = one bar, which is a per-bar projection - a 1x1 conv with a temporal receptive field of ONE BAR. It never mixed information across time, so "convolutional" described the layer type and nothing about what it computed. Same finding that sank HYBRID's LSTM. Pooling was removed on 2026-07-29 for being misconfigured against the conv output's memory layout. That removal was right; leaving the conv at a one-bar window was not. The two belong together: the NeuroNet_DNG reference (references\MQL5\Experts\EDL\Trajectory.mqh layers 2-5, kernels byte-identical to ours) pairs conv(window=2, step=1, window_out=4) with pool(window=4, step=4), and the pool only earns its place because a conv with a real receptive field sits above it. The input is bar-major (BufferTempData appends m_neuronsCount contiguous features per bar), so a flat window of k*m_neuronsCount spans exactly k bars - the receptive field needed NO kernel change. The conv output is position-major, so window == step == window_out is a clean max-over-channels, which is what the reference does and what the existing pool kernels already implement correctly. New chain at H1 defaults (420 = 20 bars x 21): conv1 w=42 s=21 out=8 -> 19 pos x 8 = 152 pool w=8 s=8 -> 19 conv2 w=2 s=1 out=8 -> 18 pos x 8 = 144 (effective field: 3 bars) We deliberately stop before the reference's SECOND pool: a channel pool emits one scalar per position, so a trailing pool would hand the dense stack 18 values and force it to fan out 18 -> 64. That is a bottleneck below every learnable layer - the same class of mistake the 2026-07-29 removal was about. Fixes a latent sizing bug this exposed: CNet's conv/pool position cursor tracked sliding POSITIONS, but a conv's real width is units_count * window_out. Any pool stacked on a conv would therefore have sized against a width window_out times too small and silently built the wrong shape. Both branches now read the built layer's actual Neurons(), which is what the batch-norm branch already did for the same reason. Also closes the architecture-pinning trap: a .nnw persists the window each conv was built with, so an existing CONV/HYBRID model would have loaded cleanly and gone on training under the OLD architecture. The conv weight tensor is (window+1)*window_out, so this cannot be repaired in place - EnforceTopologyContract now detects it, reports both shapes, and retrains. Conv chain shape is derived in one place (ConvReceptiveFieldBars / ConvFirstStagePositions / HasSecondConvStage / ConvOutputPositions / ConvOutputWidth) and consumed by AddConvStage, LstmFanIn and the startup config line, so what is built and what is logged cannot drift. Both builds compile 0 errors, 0 warnings. Forces a CONV and HYBRID retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 20:05:37 -04:00
bool m_topologySuperseded;
//--- true once ValidateCpuInference() has confirmed this model's pure-MQL5 forward pass matches
//--- the compute backend's within tolerance (see CNet::SetCpuInference). Measured at deploy on
//--- the chart (where a backend exists to compare against), never in the tester itself.
bool m_mqlInferenceValidated;
string m_fileName;
string m_folderPath;
//--- which file Train()'s Net.Save() calls (and this method's own Net.Load()) actually target:
//--- the shared FILE_COMMON production weights normally, or a LOCAL per-agent cache file when
//--- running inside the Strategy Tester/optimizer (see InitNeuralNetwork) so that repeated
//--- optimization passes with an unchanged topology can reuse an already-trained model instead of
//--- re-running every era from scratch, without ever touching the live production .nnw/.cfg.
string m_activeFileName;
bool m_activeFileCommon;
//--- Name of the terminal-wide global variable this instance holds as an exclusive claim on
//--- m_activeFileName, or "" when it holds none. See AcquireConfigLock().
string m_configLockName;
//--- user-settable via Inputs.mqh's TrainingOptimizer (SGD or ADAM), read into this member at
//--- construction.
int m_optimizationAlgo;
int m_historyBars;
//--- Input-window derivation for a NEW model (existing models adopt theirs from the .cfg):
//--- median confirmed swing leg from raw highs/lows - strict local extrema over
//--- +/-WINDOW_SWING_WING bars, alternation enforced - snapped down to {12,16,20,24,32}.
int DeriveHistoryBars(void);
int m_outputNeuronsCount;
int m_minNeuronsCount;
int m_initialNeuronsCount;
int m_neuronsCount;
double m_neuronsReduction;
int m_hiddenLayersCount;
//--- LSTM-only recurrent hidden-unit count - see LstmHiddenSize's declaration comment
//--- (Variables\Inputs.mqh). Harmless, unused constant contribution to m_fingerprint for MLP/CONV.
int m_lstmHiddenSize;
//--- CONV-only convolutional output-filter count - see ConvFilterCount's declaration comment
//--- (Variables\Inputs.mqh). Harmless, unused constant contribution to m_fingerprint for MLP/LSTM.
int m_convFilterCount;
int m_minTrainYear;
bool m_isInitialized;
fix(deinit): a full model write was running ahead of the cheap cleanup "Abnormal termination" is back, and this time it is not the arrows. The timing names the culprit exactly: 16:02:31.547 OnDeinit: shutting down 16:02:36.003 Abnormal termination <- 4.46 s, MetaTrader gave up 16:02:36.226 chart signals - persisted <- cleanup finished 0.2 s LATE OnDeinit called StopTraining() BEFORE the chart cleanup. StopTraining() finalises an in-flight run, and FinalizeTrainRun() restores the best checkpoint and then persists it - a full ~1MB model write per signal. So the expensive step ran ahead of the cheap bounded one, which is precisely the inversion the shutdown ordering exists to prevent. The previous fix put PersistWeightsOnShutdown last and missed that StopTraining smuggles a second save in at the front. Two changes: Cleanup now runs FIRST, then StopTraining, then the weight save. The visible teardown is cheap and bounded, so it always completes even when everything after it is killed. And the deploy-persist inside FinalizeTrainRun is suppressed during shutdown. RestoreWeights() is an in-MEMORY swap, so the best checkpoint is already the live net by that line, and PersistWeightsOnShutdown writes exactly those weights moments later. The old path wrote the same model twice per signal - eight full writes across four charts - for no benefit. A user-pressed Stop still persists immediately, because nothing else would. Compiles 0 errors / 0 warnings. Build tag deinit-order-v2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 16:06:40 -04:00
//--- true once OnDeinit has begun - see MarkShutdown()/FinalizeTrainRun().
bool m_shutdownInProgress;
int m_fractalPeriods;
//--- The AI's four "market models", in the role a classic signal's geometric m_pattern_N members
//--- fill: ConfidenceTierFor() buckets a fire's RAW confidence into one of four equal bands
//--- between the head's structural floor (1/3 softmax, 0.5 regression) and 1.0, and the fire
//--- votes at that tier's weight.
int m_pattern_0, m_pattern_1, m_pattern_2, m_pattern_3;
//--- TRAINING TARGET (Meta_Labeling_Design.md). Never mutated after configuration - it feeds the
//--- fingerprint like any other identity-defining member.
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
int m_trainTarget;
feat(ensemble): per-NN inputs replace the preset selector - the meta head becomes the vote's gate User design (2026-08-19): 'remove the enum menu that selects neural networks... individual inputs for every NN just like classic signals... the META NN should be integrated into the voting decision pipeline when enabled... as a bonus meta labelling is applied to enabled NNs.' - AI_CHOICE is GONE (tombstoned per the stale-.set doctrine). Use_MLP/Use_CONV/Use_LSTM/ Use_CONVLSTM are ordinary bools like the classic votes; the ensemble arithmetic adapts to any subset because the consensus divisor is the enabled capable weight. Two or more enabled = ensemble (|ENS1 token + joint gate, exactly the old AI_HYBRID fingerprints, so existing weight files keep loading); one = the old solo preset; none = classic-only. - Use_MetaLabeling un-couples META from the direction NNs (the old selector made them mutually exclusive). S3 ships: CSignalMETA::LiveMetaGate scores each vote-cleared entry (shared window at bar 1 + proposal descriptor: side, net vote, live geometry, spread/ATR; pattern one-hot ZEROED - ranking, not calibrated probability, documented in the body) and vetoes below the cost-adjusted break-even. Entries only; fail-open everywhere, loudly. - COEXISTENCE HAZARDS closed: VoteCapableWeight()=0 and ProspectiveVote()=false for the meta target - solo-only until today, a trained META would otherwise sit in the consensus divisor as a permanent abstainer and shrink every vote by its module weight. - CERTIFIED == TRADED: the ensemble era verdict replays the identical veto through the same g_warriorMetaGate pointer over its OOS fired bars (bar re-resolved from the row's own time; fail-open counted as fires and reported: 'metaGate: N approved, M vetoed, K unscored'). The overlay deliberately does NOT replay it (veto-filter-in-replay class, calendar-cliff precedent) - documented at the sweep site. Solo charts' own gate does not model the veto - the standing solo-gate caveat, documented at the input. - DB continuity: the pattern/journal DB fingerprint's first slot was (int)AIType; DbLegacyAiSlot() maps every legacy-expressible config to its OLD value (new 2-3 member subsets get 100+bitmask, outside the legacy range) so no existing database re-keys. filterID becomes the enabled roster via one EnabledNNSummary(). - HUD: the meta line shows the gate (armed/(trn), last P vs BE, ok/veto tally); the armed/disarmed announcement fires on state change via one latch (MetaGateArmedNow), not only when an entry happens to be proposed. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 13:01:02 -04:00
//--- True when this signal runs as one of the ensemble's members - see EnsembleMember().
bool m_ensembleMember;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
//--- Slot in g_warriorEnsemble (registration order, -1 = not an ensemble member). Doubles as the
//--- bit index in the combined-vote masks and the cursor index - see the registry's header comment.
int m_ensembleIndex;
2026-08-15 19:00:40 -04:00
//--- CONDITIONAL leg excursions for the fractal target's geometry (user request 2026-08-15:
//--- "calculate MAE and MFE from a fractal to the next"). Safe against circularity ONLY because
//--- the fractal label does not depend on SL/TP (the barrier label does - never feed it this
//--- path).
2026-08-15 19:00:40 -04:00
double m_fracLegFav[];
double m_fracLegAdv[];
int m_fracLegCount;
void RecordFractalLegExcursion(const double fav, const double adv)
{
if(fav <= 0.0 && adv <= 0.0)
return;
int cap = ArraySize(m_fracLegFav);
if(m_fracLegCount >= cap)
{
cap += cap / 2 + 256;
ArrayResize(m_fracLegFav, cap);
ArrayResize(m_fracLegAdv, cap);
}
m_fracLegFav[m_fracLegCount] = MathMax(fav, 0.0);
m_fracLegAdv[m_fracLegCount] = MathMax(adv, 0.0);
m_fracLegCount++;
}
//--- This member's slot in the combined ensemble panel; claimed lazily on first publish (-1 = none).
int m_ensemblePanelSlot;
//--- Meta candidate store for the CURRENT era's bar grid, populated by MetaPrepareEra()
//--- (overridden in CSignalMETA; empty and unused for direction models).
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
int m_metaCandBar[]; // series index of the candidate's fire bar
char m_metaCandSide[]; // +1 long / -1 short
double m_metaCandNetVote[]; // the firing filter's own vote margin (raw weight units)
short m_metaCandFamily[]; // 0=MA 1=RSI 2=MACD 3=Ichimoku
short m_metaCandPattern[]; // Pattern_N within the family
int m_metaCandCount;
//--- per-bar chain: m_metaCandHead[barIdx] -> first candidate id at that bar (-1 none),
//--- m_metaCandNext[candId] -> next candidate at the same bar. Rebuilt with the store.
int m_metaCandHead[];
int m_metaCandNext[];
//--- Candidate id for each pass-2 queue slot, parallel to m_isTrainQueue (see Training.mqh's
//--- queueing block); -1 on every slot for direction models. Swapped in lockstep by the shuffle.
int m_isTrainQueueCand[];
//--- Per-family (0-3) and per-side (0=long 1=short) OOS decomposition of the meta head's era -
//--- candidates / base wins / operating-point trades / wins among trades. Reset each era beside
//--- m_oosBuyFired.
int m_metaFamCand[4], m_metaFamWins[4], m_metaFamFired[4], m_metaFamFiredWins[4];
int m_metaSideCand[2], m_metaSideWins[2], m_metaSideFired[2], m_metaSideFiredWins[2];
//--- functions Creates the OHLC + ZigZag indicators the feature builder reads. Called by
//--- InitNeuralNetwork(), never by the framework - the PUBLIC InitIndicators() override below is
//--- the framework entry point.
refactor(signals): AI signal files are identity + topology, nothing else Every AI signal repeated the same five-line InitIndicators override that did nothing but call InitNeuralNetwork. The cause was an access mismatch, not a design: CExpertSignalCustom declares InitIndicators public, the AI base redeclared it PROTECTED, and each subclass had to redeclare it public to be reachable by CExpert. Worse, the base's own override does a different job entirely - it creates the OHLC/ZigZag feature indicators - and InitNeuralNetwork called it back scope-qualified to stop the virtual dispatch landing in the subclass. Two jobs, one virtual name, and a recursion trap held off by a scope qualifier. The feature-indicator step is now InitFeatureIndicators() (protected, non-virtual, named for what it does) and the AI base carries the single public InitIndicators override. CONV/HYBRID/LSTM/PAI/META drop their copies and are now purely identity plus topology, which is the classic signal file's shape. Comment pass on ExpertSignalAIBase.mqh, -100 lines with every constant and every measured number kept. Three claims in the tier block were stale and inverted - it named CalibratedConfidenceMagnitude() as the tiering input where the code deliberately uses the RAW magnitude, and it described the signal DB as re-ranking each tier when ApplyPatternWeight declines the DB from the end of era 1. Also dropped a paragraph whose subject was a previous version of the comment, and moved two notes down onto the constants they document (CONV_COMPRESSION_DIVISOR was 16 lines and three unrelated defines away from its own text). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 08:57:54 -04:00
bool InitFeatureIndicators(CIndicators *indicators);
//--- sets ID/m_id/m_folderPath/m_fileName/m_pattern_count from the subclass constructor - defaults
//--- to 4 (the confidence tiers - see m_pattern_0's declaration comment), not 1
void SetIdentity(string id, string shortId, int patternCount = 4);
//--- hook for neuron-type-specific layers (Conv+Pool, LSTM, ...); default is a plain perceptron (no-op)
virtual bool AddCustomLayers(CArrayObj *topology) { return true; }
//--- Reusable front-end stages, composed by the AddCustomLayers() overrides. (They had already
//--- drifted - HYBRID guarded the LSTM step with MathMax(1,...) and CSignalLSTM did not, so a
//--- historyBars of 1 gave the two a different step.)
fix(ai): drop the conv pooling stage - it reduced across filters, not time FeedForwardConv emits POSITION-MAJOR output, matrix_o[out + window_out * i], so one bar's window_out filter responses are contiguous and consecutive bars sit window_out apart. Both pooling implementations (FeedForwardProof and CPU_FeedForwardProof) slide FLAT over that buffer - pos = i * step, reducing `window` CONSECUTIVE elements. On a position-major layout those neighbours are different FILTERS of the same bar, never one filter across time. At the shipped 3/2 the pool computed max(bar0_f0, bar0_f1, bar0_f2), then max(bar0_f2, bar0_f3, bar0_f4), with every 8th window straddling a bar boundary. So it collapsed unrelated feature detectors into whichever fired hardest, passed gradient to that winner only, and halved the feature map while doing it - all below every learnable layer, where nothing above can recover it. The removed inputs' own labels ("3 Bars") show time-axis pooling was the intent throughout. Measured cost: CONV sat pinned at ~40% balanced accuracy for 510 eras with Sell recall 0%, while plain MLPs on the same data reached 57-61%. HYBRID, which also carried this stage, came second-worst of the batch-norm group. Not fixable in the topology: pooling one filter across time needs a stride of window_out BETWEEN samples within a window, which a consecutive-window kernel cannot express at any window/step. That needs a stride-aware kernel in Network.cl + WarriorCPU.cpp + WarriorDML.cpp and a DLL rebuild, and is only worth doing if a conv front-end earns its place without downsampling first - with 20 sliding positions there is little to gain by halving them. ConvPoolWindow/ConvPoolStep and their enums are removed with it, along with the |CP: fingerprint term added earlier today. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:28:44 -04:00
bool AddConvStage(CArrayObj *topology);
refactor: compose topologies from named stages; drop dead code DRY - topology construction --------------------------- CSignalCONV and CSignalHYBRID each built the Conv+Pool front-end from scratch; CSignalLSTM and CSignalHYBRID each built the LSTM stage from scratch. The duplicates had already drifted: HYBRID guarded the LSTM step with MathMax(1, historyBars/2), CSignalLSTM divided unguarded, so a historyBars of 1 gave two different steps for what is documented as the same layer. Extracted AddConvPoolStage() and AddLstmStage() onto CExpertSignalAIBase. The three overrides are now compositions: CONV = AddConvPoolStage LSTM = AddLstmStage HYBRID = AddConvPoolStage && AddLstmStage HYBRID's "matches the standalone CONV front-end exactly, then adds LSTM" is enforced by construction instead of by comment. Took the guarded step for both. Also fixed a descriptor leak the duplicates shared: on a failed topology.Add() the CLayerDescription was neither owned by the array nor deleted. Dead code --------- - CNet::SaveCheckpoint / CNet::LoadCheckpoint (123 lines). Superseded by the in-memory CaptureWeights/RestoreWeights pair; Network.mqh:1312 already said so ("This replaces the file-based SaveCheckpoint/LoadCheckpoint"). Zero call sites - every remaining mention was a comment. The five comments that referenced them have been reworded rather than left dangling. - CExpertSignalCustom::CheckForDuplicateTrade / FindLastTradeIndex / UpdateTradeStatusAndExit: declared, never defined anywhere, never called. They only made it look as though duplicate-trade detection existed. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:38:05 -04:00
bool AddLstmStage(CArrayObj *topology);
//--- Which front-end stages this subclass's AddCustomLayers() actually appends. A virtual rather
//--- than a type-enum check, so a future composition cannot silently get the wrong answer.
virtual bool UsesConvStage(void) const { return false; }
virtual bool UsesLstmStage(void) const { return false; }
//--- META-TARGET SEAMS (all no-ops for direction models; overridden only by CSignalMETA). The
//--- rest of the seams stay here. FRACTAL TARGET: direction to the next confirmed fractal
//--- extreme on every bar, ~balanced by construction.
feat(ai): TrainingTarget input - fractal-direction label for the direction models User direction (2026-08-15): back to predicting swing turns, D1 charts, fractals over ZigZag pivots (their call - balances classes, matches the reference library target, and a 5-bar fractal confirms 2 bars after its extreme so labels resolve nearly to the present with no repaint embargo). - TRAINING_TARGET enum + TrainingTarget input: TARGET_BARRIER (Market default - existing models keep their meaning and fingerprints) or TARGET_FRACTAL (private default). - FractalDirectionLabel (Labels.mqh): per-bar 3-class label = direction from the bar close to the next confirmed strict 5-bar fractal extreme, costs charged in the same bid-series convention as the barrier label, Neutral when the move cannot clear max(2 spreads, 0.10 ATR) or on an outside bar (both-extreme bars are unorderable within OHLC). - The barrier walk still runs in full: measured SL/TP geometry, the expectancy scan, excursion caches and the era gate all keep scoring what a trade at the EA's own stop/target actually collected - only the TRAINING label changes. NOT the pre-b4a704d "is this bar the pivot" form; that target's 31:1 imbalance stays retired. - Fingerprint token |TGT:FRA1 so switching targets trains a separate model; AI_META unaffected (guarded setter). - Private defaults: AIType back to AI_HYBRID (direction topology needed) + TrainingTarget=TARGET_FRACTAL = drop-on-D1-chart workflow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 04:44:10 -04:00
bool IsFractalTarget(void) const { return m_trainTarget == 2; }
//--- Labels.mqh: fractal-direction label for one bar (overrides the barrier verdict in
//--- AdvanceBarrierLabelState when IsFractalTarget()).
ENUM_SIGNAL FractalDirectionLabel(int idx);
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- Resolve the candidate corpus onto this era's bar grid (fills the m_metaCand* store). Called at
//--- every era start, right after the bar grid is sized; returning false aborts the training run.
virtual bool MetaPrepareEra(const int bars) { return true; }
//--- Append the per-candidate setup descriptor to TempData, AFTER BuildFeatureWindow() has filled
//--- the shared bar window. The input layer is sized historyBars*features + MetaDescWidth(), so
//--- every feedForward on a meta net MUST run this between window build and forward.
virtual void AppendCandidateFeatures(const int candId) {}
//--- Width of that descriptor; 0 for direction models so NetInputWidth() stays byte-identical.
virtual int MetaDescWidth(void) const { return 0; }
//--- The one true input width every feedForward guard compares against.
int NetInputWidth(void) const { return (int)m_historyBars * m_neuronsCount + MetaDescWidth(); }
//--- P(win) from the 2-output head's raw activations in TempData (after Net.getResults) - the
//--- 2-class softmax collapses to a logistic over the logit difference. Same CLASS_LOGIT_SCALE the
//--- training gradient applies, so the probability is the one the loss was optimizing. -1 = no data.
double MetaWinProbability(void)
{
if(TempData.Total() < 2)
return -1.0;
double z = CLASS_LOGIT_SCALE * (TempData.At(0) - TempData.At(1));
return 1.0 / (1.0 + MathExp(-z));
}
//--- Triple-barrier outcome of the candidate's own side at its fire bar - the meta LABEL. Reads the
//--- side-conditional win caches the label prebuild already computes for every bar; loss AND
//--- timeout are both 0, matching the design ("win=1 / loss-or-timeout=0").
bool MetaCandidateWon(const int candId, const int barIdx)
{
if(barIdx < 0 || barIdx >= ArraySize(m_labelCacheHasValue) || !m_labelCacheHasValue[barIdx])
return false;
if(m_metaCandSide[candId] > 0)
return (barIdx < ArraySize(m_winLongCache)) ? m_winLongCache[barIdx] : false;
return (barIdx < ArraySize(m_winShortCache)) ? m_winShortCache[barIdx] : false;
}
//--- First candidate id at a bar (-1 none) / next in the same-bar chain.
int MetaCandFirst(const int barIdx) const
{ return (barIdx >= 0 && barIdx < ArraySize(m_metaCandHead)) ? m_metaCandHead[barIdx] : -1; }
int MetaCandNext(const int candId) const
{ return (candId >= 0 && candId < ArraySize(m_metaCandNext)) ? m_metaCandNext[candId] : -1; }
//--- AddConvStage runs BEFORE AddLstmStage wherever both are present (HYBRID), so the LSTM is fed the
//--- conv feature map rather than the raw flattened input.
bool HasConvBeforeLstm(void) const { return UsesConvStage() && UsesLstmStage(); }
feat(ai): real conv receptive field + the reference's channel pool CONV's convolution used window = step = one bar, which is a per-bar projection - a 1x1 conv with a temporal receptive field of ONE BAR. It never mixed information across time, so "convolutional" described the layer type and nothing about what it computed. Same finding that sank HYBRID's LSTM. Pooling was removed on 2026-07-29 for being misconfigured against the conv output's memory layout. That removal was right; leaving the conv at a one-bar window was not. The two belong together: the NeuroNet_DNG reference (references\MQL5\Experts\EDL\Trajectory.mqh layers 2-5, kernels byte-identical to ours) pairs conv(window=2, step=1, window_out=4) with pool(window=4, step=4), and the pool only earns its place because a conv with a real receptive field sits above it. The input is bar-major (BufferTempData appends m_neuronsCount contiguous features per bar), so a flat window of k*m_neuronsCount spans exactly k bars - the receptive field needed NO kernel change. The conv output is position-major, so window == step == window_out is a clean max-over-channels, which is what the reference does and what the existing pool kernels already implement correctly. New chain at H1 defaults (420 = 20 bars x 21): conv1 w=42 s=21 out=8 -> 19 pos x 8 = 152 pool w=8 s=8 -> 19 conv2 w=2 s=1 out=8 -> 18 pos x 8 = 144 (effective field: 3 bars) We deliberately stop before the reference's SECOND pool: a channel pool emits one scalar per position, so a trailing pool would hand the dense stack 18 values and force it to fan out 18 -> 64. That is a bottleneck below every learnable layer - the same class of mistake the 2026-07-29 removal was about. Fixes a latent sizing bug this exposed: CNet's conv/pool position cursor tracked sliding POSITIONS, but a conv's real width is units_count * window_out. Any pool stacked on a conv would therefore have sized against a width window_out times too small and silently built the wrong shape. Both branches now read the built layer's actual Neurons(), which is what the batch-norm branch already did for the same reason. Also closes the architecture-pinning trap: a .nnw persists the window each conv was built with, so an existing CONV/HYBRID model would have loaded cleanly and gone on training under the OLD architecture. The conv weight tensor is (window+1)*window_out, so this cannot be repaired in place - EnforceTopologyContract now detects it, reports both shapes, and retrains. Conv chain shape is derived in one place (ConvReceptiveFieldBars / ConvFirstStagePositions / HasSecondConvStage / ConvOutputPositions / ConvOutputWidth) and consumed by AddConvStage, LstmFanIn and the startup config line, so what is built and what is logged cannot drift. Both builds compile 0 errors, 0 warnings. Forces a CONV and HYBRID retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 20:05:37 -04:00
//--- Conv chain shape - see the definitions above AddConvStage. Every consumer reads these rather
//--- than re-deriving the arithmetic, so the built topology and the logged shape cannot disagree.
int ConvReceptiveFieldBars(void) const;
int ConvFirstStagePositions(void) const;
bool HasSecondConvStage(void) const;
int ConvOutputPositions(void) const;
int ConvOutputWidth(void) const;
fix(ai): stop the shutdown save from resurrecting reset weights; size HYBRID's LSTM to its real fan-in ResetWeights already deletes the whole model set - .nnw, .cfg, _ckpt.tmp, .stats, _shadow.nnw - and clears both the .arrows sidecar and the drawn chart objects. What undid it was PersistWeightsOnShutdown: detaching the EA after a reset but before an era completed re-created a .nnw from the freshly-built, never-run net, so the next attach loaded an era-0 stub instead of starting clean. For LSTM/HYBRID that stub is worse than nothing - a layer that has never run a forward pass has m_iInputs<=0, so Save omits every LSTM buffer (see 413ff7e). Skip the save when no era completed and no model was loaded; that is exactly the post-reset and first-attach state. Also sweep _shadowclone.tmp, which the reset did not cover. Separately, ComputeLstmHiddenSize budgeted every topology against the flattened input (historyBars x neuronsCount). True for LSTM, wrong for HYBRID, where AddConvStage runs first and the LSTM is fed the conv feature map - historyBars x convFilterCount, 160 rather than 420 at H1 defaults. The quadratic is dominated by the inputs term, so overstating the fan-in 2.6x cost a full ladder step (16 units where the budget affords 32). New virtual HasConvBeforeLstm() feeds LstmFanIn(), so composition decides this rather than an AIType check. desc.window is advisory only - CNet never passes it to the layer - but is now truthful for the same reason. Derived values stay out of the weights-filename fingerprint and are adopted from the .cfg, so existing models keep their saved width; only fresh ones pick up the corrected budget. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 13:06:09 -04:00
//--- Actual input width the LSTM block sees, which is NOT always the flattened input.
int LstmFanIn(void) const;
//--- " | conv 21->8 x20 bars | lstm 160->32" for the startup config line; "" when neither applies.
string FrontEndConfigSummary(void) const;
feat(ai): batch normalization between dense layers The only bounded stage in the entire forward path was the sigmoid classification head - every hidden stage is PRELU. That is a network with no internal scale control, and the failure ordered exactly by depth: on SP500 H1 the shallow perceptron held ~52% balanced accuracy while the deepest topology sat on the 33.3% one-class floor, with the per-bar logit spread decaying monotonically (0.45 -> 0.38 over ~200 eras) until the evidence tilt fell under the class-prior tilt. That is the signature of internal covariate shift, which chapter 6.1 of the reference book is entirely about and which the NeuroNet_DNG engine addresses with a layer this project never had. Two mechanisms make this the right fix rather than more hyperparameter nudging: - it decouples WEIGHT_DECAY from the learned function (van Laarhoven 2017) - with a normalized layer downstream, decay can no longer grind the discriminative signal away, it only rescales the effective learning rate; - it is the precondition for ever running an unbounded logit head here. The 2026-07-27 attempt blew up (IS error 5.6e15) precisely because nothing upstream constrained scale. Implementation notes: - CNeuronBatchNormOCL computes host-side rather than as a fourth copy of a kernel across Network.cl + WarriorCPU.cpp + WarriorDML.cpp. The math is elementwise O(n); this way it behaves identically on all four compute tiers, needs no DLL rebuild, and cannot drift between backends. Same precedent as the softmax+CCE gradient and the per-sample loss weighting, both computed in MQL5 for that reason. - Statistics are exponential moving, not a stored mini-batch: training is pure online SGD, one update per sample, so there is no batch to average over. BatchNormWindow is an EMA window length. - gamma/beta are excluded from weight decay, deliberately - decaying gamma toward zero is the exact pathology being fixed. - The layer self-sizes from whatever sits below it, because a conv/pool stage's output width is derived inside the CNet constructor and is not knowable to the topology builder. - Checkpoint capture/restore/blend carry gamma/beta and the running statistics alongside the dense matrix, so the plateau ladder cannot restore a mismatched pair. - SeedOutputLayerBias accepted only an exact defNeuronBaseOCL as the weight-carrying penultimate layer; with normalization enabled that is the batch-norm layer, so the cold-start bias seed would have silently stopped being applied. - Refuses to build, loudly, if a topology asks for normalization with no compute backend at all - rather than quietly training a different architecture than the one requested. EnableBatchNorm (default on) and BatchNormWindow (1000 samples) are inputs so the effect can be A/B'd without a recompile. Both feed the weights-filename fingerprint, appended conditionally so existing non-BN configs keep their fingerprints and are not forced to retrain. Verified: analytic gradients match finite differences to 1.5e-7 relative over 200 random cases; a faithful port of the full forward/backward chain collapses to the 33.3% floor by era 4 without this layer and holds 36-43% with it. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 12:34:29 -04:00
//--- Appends a batch-normalization layer, or does nothing (returning success) when EnableBatchNorm is
//--- off. `units` is advisory only - CNet sizes the layer from whatever sits below it, because a conv
//--- or pool stage's output width is derived inside the CNet constructor and is not knowable here.
//--- See AI\NeuronBatchNorm.mqh for what the layer does and why it exists.
bool AddBatchNormStage(CArrayObj *topology, int units);
//--- hardcoded activation for the common tapering Dense hidden-layer stack built by
//--- BuildFreshTopology() (below AddCustomLayers, above the output layer).
virtual ENUM_ACTIVATION HiddenLayerActivation(void) { return PRELU; }
//--- Single source of truth for the output head's activation. As two separate literals, changing
//--- the head silently did nothing to any existing model. Regression: TANH, whose [-1,1] maps
//--- onto the Sell/Neutral/Buy convention.
fix: stop a .nnw from pinning a superseded architecture A .nnw persists the ARCHITECTURE, not just the weights: Save writes (int)activation per neuron and Load reads it straight back. The activation chosen in BuildFreshTopology() therefore only ever reached a brand-new topology - every reload restored the file's value and the next save wrote it back out, so a wrong value could never heal while the source read as though it were already fixed. That is how five models kept training with an unbounded NONE classification head for a full day after the 07-28 revert to SIGMOID. Confirmed by parsing the binaries: 848cb42c.nnw / 2e754b43.nnw carry `act=NONE` on the 3-neuron output layer, while a genuinely reset model of the same config carries act=SIGMOID. In the log it showed as negative "OOS raw out" values - impossible under sigmoid - escalating to a 4.14e13 logit spread with all three classes numerically identical (input-independent output) and balanced accuracy pinned on the 33.3% one-class floor. - OutputLayerActivation() is now the single source of truth, called by both BuildFreshTopology() and the new load-time repair, so the two can no longer diverge the way a duplicated literal did. - CNet::EnforceOutputActivation() re-asserts it after Load and reports the stale value; CExpertSignalAIBase::EnforceTopologyContract() logs the repair loudly, since weights learned under the old head may not be worth keeping even once the head is corrected. - Hidden layers are deliberately left alone: they legitimately differ per stage (PRELU dense/conv, NONE pool, TANH LSTM). Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 12:00:40 -04:00
ENUM_ACTIVATION OutputLayerActivation(void) const { return (m_outputNeuronsCount == 1) ? TANH : SIGMOID; }
//--- Width of the first dense layer, DERIVED rather than configured. That is ~8 parameters per
//--- sample, and it EXPANDS a set of highly correlated inputs instead of compressing them. MUST
//--- be called before the fingerprint is built and never again (see the note on fingerprint-
//--- feeding members at the top of this file).
refactor(ai): derive the first dense layer's width instead of asking for it InitialNeurons was an input whose only defensible value depends on two things the user cannot see when picking from a dropdown: how wide the input vector ended up after feature selection, and how much in-sample data the study period actually yields. Left to a hand-picked constant it was badly wrong - 500 units against a 420-wide input is 210,500 weights, 72% of a 292,583-weight model, against ~36,500 training bars of which only ~2,236 are directional. That is 6.6 weights per training bar, and it EXPANDS a set of highly correlated inputs rather than compressing them. The symptom was already in the logs and had been read as a depth problem: the shallowest topology consistently beat the deepest (perceptron 52.7% balanced, hybrid 41.3%). Over-parameterization predicts that ordering just as well as covariate shift does, and only one of the two had been addressed. ComputeFirstLayerWidth() budgets roughly one first-layer weight per in-sample bar. Measured across the configurations in use: M15 10y -> 256 units, 129,071 weights, 0.73 per bar H1 10y -> 64 units, 28,727 weights, 0.65 per bar H4 10y -> 16 units, 7,559 weights, 0.68 per bar Two design points that matter: - It estimates in-sample bars from the STUDY PERIOD and timeframe, not from Bars(). What is downloaded grows over a terminal's lifetime, and a topology that widened as history filled in would re-key its own weights file and discard a trained model. - The result is snapped down to a coarse power-of-two ladder, so the estimate would have to be wrong by ~2x to change the answer. Every field it reads is already part of the weights-filename fingerprint, so the derived value needs no fingerprint entry of its own. The public setter is removed - it could only have been called after construction, and would either be ignored or silently re-key the model mid-run. Where the data cannot support even the floor (D1 over 10 years is under 2,000 bars) it now says so and names the fixes, rather than quietly training a model with more weights than examples. The DB config fingerprint drops the term too, which re-keys existing pattern databases once - correct, since a model an order of magnitude smaller should not inherit the old one's win-rate history. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 13:01:16 -04:00
int ComputeFirstLayerWidth(void) const;
fix(topology): the capacity budget counted overlapping bars as independent examples EstimatedInSampleBars() returned raw bars (11372 on SP500 H4) and every derived capacity decision spent that: first-layer width, conv filters, LSTM hidden size. But triple-barrier labels overlap - mean lifespan 9.4 bars - so the label cache line on the same run already reports those bars are worth ~1210 independent observations. Sizing a network against RAW bars while grading it against EFFECTIVE ones is two subsystems disagreeing about one sample, and it disagreed in the dangerous direction because the capacity side was the optimistic one: the warning's "roughly 1.1 weights per training bar" is nearer 11 per independent observation. EffectiveSampleSize() has existed since 2026-08-17 and is applied at eight sites, all of them statistics. This adds the ninth, in the one place that decides how many parameters get fitted. Applied inside EstimatedInSampleBars() rather than at the call sites, because that function exists precisely so the three stages spend one budget. SELF-ENABLING AND THEREFORE INERT WHERE IT MATTERS MOST, which is why this is two changes and not one. MeanLabelLifespan() is 1.0 until a label cache has measured something, so on a model's first build - before any label exists - the deflation is correctly the identity: an unmeasured overlap must not invent a shrink. A fresh attach constructs a fresh object, so its counters are zero too; only a mid-session weights reset carries real evidence into a rebuild. That is deliberately safe (no attach can now re-derive a narrower topology and discard trained weights) but it would have left the first build - the case you most want the truth for - quoting the flattering figure. So ReportDetectability now restates capacity against the effective sample at the first moment L is real, for the topology already pinned. It re-sizes nothing; it reports what was bought. Placed ABOVE that function's break-even guard on purpose - a degenerate geometry is exactly when you want to know the net is over-parameterised, and "it only fires for sane configs" is how the 2026-08-18 IS-error stop managed never to fire at all. The warning also names its basis now (independent observations and L, or an explicit "overlap NOT YET MEASURED, this is an UPPER BOUND"), so a flattering number can never again read as a measured one. Also factors FirstLayerFanIn() out of ComputeFirstLayerWidth so the capacity REPORT charges for exactly what the capacity DECISION charged for - same reason RequiredHorizonBars was factored out after the 2026-08-17 divergence - and makes MeanLabelLifespan()/EffectiveSampleSize() const so the const budget path can call them. Verified: no recursion (EstimatedInSampleBars -> EffectiveSampleSize -> EstimatedInSampleBarsRaw, which computes from Bars() alone); both new StringFormat sites hand-counted (basis 3/3 and 1/1, CAPACITY 10 specifiers / 10 arguments). NOT COMPILED - user compiles in MetaEditor.
2026-08-19 18:43:48 -04:00
//--- Expected in-sample training rows for the configured study period, split and timeframe, in
//--- INDEPENDENT observations. See the definition for why a raw bar count was the wrong unit to
//--- size a network in.
feat(nn): derive conv filter count and LSTM hidden size from the data Same defect the first-layer width had before 2026-07-29: both were inputs whose defaults were fixed constants picked with no reference to the input they sit on, which is the only thing that decides whether either number is sane. The conv layer is a per-bar projection - AddConvStage sets window = step = one bar's features - so its filter count should be read against the per-bar feature count. Sixteen filters COMPRESSED a 50-feature configuration 3x but EXPANDED a minimal 4-feature one 4x, and the expanding case adds parameters below every learnable layer without adding information. Now derived as half the per-bar feature count, snapped down a power-of-two ladder. The LSTM stage was the bigger miss. Its weight count is exactly 4*H*(H+inputs+1) (CNeuronLSTMOCL::SetInputs) and AddLstmStage feeds it the whole flattened vector, so the shipped 32 units against a 540-wide input is ~73k weights - more than DOUBLE the entire derived dense taper it feeds. It was the one stage the capacity budget never covered, which is why deriving the dense stack alone did not stop LSTM and HYBRID from being over-parameterized. Now solved from the same one-weight-per-in-sample-bar budget the first layer spends. Factored EstimatedInSampleBars() out of ComputeFirstLayerWidth so all three decisions spend one budget rather than each guessing at the training-set size separately. Both new values are assigned alongside the first-layer width, before the fingerprint that hashes them, and are functions of inputs already in that hash - so they need no entry of their own, and the same reasoning removes them from the DB config key. Both builds compile 0 errors, 0 warnings. Re-keys existing models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 09:22:11 -04:00
double EstimatedInSampleBars(void) const;
fix(topology): the capacity budget counted overlapping bars as independent examples EstimatedInSampleBars() returned raw bars (11372 on SP500 H4) and every derived capacity decision spent that: first-layer width, conv filters, LSTM hidden size. But triple-barrier labels overlap - mean lifespan 9.4 bars - so the label cache line on the same run already reports those bars are worth ~1210 independent observations. Sizing a network against RAW bars while grading it against EFFECTIVE ones is two subsystems disagreeing about one sample, and it disagreed in the dangerous direction because the capacity side was the optimistic one: the warning's "roughly 1.1 weights per training bar" is nearer 11 per independent observation. EffectiveSampleSize() has existed since 2026-08-17 and is applied at eight sites, all of them statistics. This adds the ninth, in the one place that decides how many parameters get fitted. Applied inside EstimatedInSampleBars() rather than at the call sites, because that function exists precisely so the three stages spend one budget. SELF-ENABLING AND THEREFORE INERT WHERE IT MATTERS MOST, which is why this is two changes and not one. MeanLabelLifespan() is 1.0 until a label cache has measured something, so on a model's first build - before any label exists - the deflation is correctly the identity: an unmeasured overlap must not invent a shrink. A fresh attach constructs a fresh object, so its counters are zero too; only a mid-session weights reset carries real evidence into a rebuild. That is deliberately safe (no attach can now re-derive a narrower topology and discard trained weights) but it would have left the first build - the case you most want the truth for - quoting the flattering figure. So ReportDetectability now restates capacity against the effective sample at the first moment L is real, for the topology already pinned. It re-sizes nothing; it reports what was bought. Placed ABOVE that function's break-even guard on purpose - a degenerate geometry is exactly when you want to know the net is over-parameterised, and "it only fires for sane configs" is how the 2026-08-18 IS-error stop managed never to fire at all. The warning also names its basis now (independent observations and L, or an explicit "overlap NOT YET MEASURED, this is an UPPER BOUND"), so a flattering number can never again read as a measured one. Also factors FirstLayerFanIn() out of ComputeFirstLayerWidth so the capacity REPORT charges for exactly what the capacity DECISION charged for - same reason RequiredHorizonBars was factored out after the 2026-08-17 divergence - and makes MeanLabelLifespan()/EffectiveSampleSize() const so the const budget path can call them. Verified: no recursion (EstimatedInSampleBars -> EffectiveSampleSize -> EstimatedInSampleBarsRaw, which computes from Bars() alone); both new StringFormat sites hand-counted (basis 3/3 and 1/1, CAPACITY 10 specifiers / 10 arguments). NOT COMPILED - user compiles in MetaEditor.
2026-08-19 18:43:48 -04:00
//--- The same figure BEFORE the overlap deflation, for reports that want to show both. Never size
//--- anything from this one - that was the bug.
double EstimatedInSampleBarsRaw(void) const;
//--- Width of the vector the first dense layer actually sees: the front-end stage's output where
//--- one exists, the flattened window otherwise.
fix(topology): the capacity budget counted overlapping bars as independent examples EstimatedInSampleBars() returned raw bars (11372 on SP500 H4) and every derived capacity decision spent that: first-layer width, conv filters, LSTM hidden size. But triple-barrier labels overlap - mean lifespan 9.4 bars - so the label cache line on the same run already reports those bars are worth ~1210 independent observations. Sizing a network against RAW bars while grading it against EFFECTIVE ones is two subsystems disagreeing about one sample, and it disagreed in the dangerous direction because the capacity side was the optimistic one: the warning's "roughly 1.1 weights per training bar" is nearer 11 per independent observation. EffectiveSampleSize() has existed since 2026-08-17 and is applied at eight sites, all of them statistics. This adds the ninth, in the one place that decides how many parameters get fitted. Applied inside EstimatedInSampleBars() rather than at the call sites, because that function exists precisely so the three stages spend one budget. SELF-ENABLING AND THEREFORE INERT WHERE IT MATTERS MOST, which is why this is two changes and not one. MeanLabelLifespan() is 1.0 until a label cache has measured something, so on a model's first build - before any label exists - the deflation is correctly the identity: an unmeasured overlap must not invent a shrink. A fresh attach constructs a fresh object, so its counters are zero too; only a mid-session weights reset carries real evidence into a rebuild. That is deliberately safe (no attach can now re-derive a narrower topology and discard trained weights) but it would have left the first build - the case you most want the truth for - quoting the flattering figure. So ReportDetectability now restates capacity against the effective sample at the first moment L is real, for the topology already pinned. It re-sizes nothing; it reports what was bought. Placed ABOVE that function's break-even guard on purpose - a degenerate geometry is exactly when you want to know the net is over-parameterised, and "it only fires for sane configs" is how the 2026-08-18 IS-error stop managed never to fire at all. The warning also names its basis now (independent observations and L, or an explicit "overlap NOT YET MEASURED, this is an UPPER BOUND"), so a flattering number can never again read as a measured one. Also factors FirstLayerFanIn() out of ComputeFirstLayerWidth so the capacity REPORT charges for exactly what the capacity DECISION charged for - same reason RequiredHorizonBars was factored out after the 2026-08-17 divergence - and makes MeanLabelLifespan()/EffectiveSampleSize() const so the const budget path can call them. Verified: no recursion (EstimatedInSampleBars -> EffectiveSampleSize -> EstimatedInSampleBarsRaw, which computes from Bars() alone); both new StringFormat sites hand-counted (basis 3/3 and 1/1, CAPACITY 10 specifiers / 10 arguments). NOT COMPILED - user compiles in MetaEditor.
2026-08-19 18:43:48 -04:00
int FirstLayerFanIn(void) const;
//--- Conv output-filter count and LSTM hidden width, DERIVED for the same reason the first-layer
//--- width is. Both MUST be called before the fingerprint is built and never again: they assign
//--- fingerprint-feeding members (see the note at the top of this file).
feat(nn): derive conv filter count and LSTM hidden size from the data Same defect the first-layer width had before 2026-07-29: both were inputs whose defaults were fixed constants picked with no reference to the input they sit on, which is the only thing that decides whether either number is sane. The conv layer is a per-bar projection - AddConvStage sets window = step = one bar's features - so its filter count should be read against the per-bar feature count. Sixteen filters COMPRESSED a 50-feature configuration 3x but EXPANDED a minimal 4-feature one 4x, and the expanding case adds parameters below every learnable layer without adding information. Now derived as half the per-bar feature count, snapped down a power-of-two ladder. The LSTM stage was the bigger miss. Its weight count is exactly 4*H*(H+inputs+1) (CNeuronLSTMOCL::SetInputs) and AddLstmStage feeds it the whole flattened vector, so the shipped 32 units against a 540-wide input is ~73k weights - more than DOUBLE the entire derived dense taper it feeds. It was the one stage the capacity budget never covered, which is why deriving the dense stack alone did not stop LSTM and HYBRID from being over-parameterized. Now solved from the same one-weight-per-in-sample-bar budget the first layer spends. Factored EstimatedInSampleBars() out of ComputeFirstLayerWidth so all three decisions spend one budget rather than each guessing at the training-set size separately. Both new values are assigned alongside the first-layer width, before the fingerprint that hashes them, and are functions of inputs already in that hash - so they need no entry of their own, and the same reasoning removes them from the DB config key. Both builds compile 0 errors, 0 warnings. Re-keys existing models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 09:22:11 -04:00
int ComputeConvFilterCount(void) const;
int ComputeLstmHiddenSize(void) const;
//--- Dense-taper DEPTH, derived 2026-07-30 from the two endpoints the taper connects. Reads
feat(nn): derive dense depth, train on all history, pin the shape in .cfg Completes the derived-topology work. Three inputs removed. AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five entries instead of eight. Depth is now derived from the two endpoints the taper already has to connect (derived first-layer width, output-tied final width) at a 2x per-layer compression target, clamped [2..5]. Asking a user to pick a layer count while the code derives the widths those layers taper between was asking for half a decision: at 64 units tapering to 12, four layers compress by 1.4x per step and five by 1.3x, so the extra depth bought no abstraction. On the shipping H1/10y default the derivation lands on 3 layers - the depth that actually won Run 2. StudyPeriods removed. There is no case for training on less data than the broker provides at a ~6% directional base rate; the honest generalization read comes from the OOS holdout, not from withholding history. Training now starts at the earliest available bar, floored by MinTrainYear, which answers a different question (excluding dubious pre-history) and stays. That required closing the hazard the old code documented: the capacity budget now MEASURES the symbol's real bar count, and a topology derived from a measurement would widen as history downloads. Both ends are now pinned. Every derived value left the weights-filename fingerprint - keying a filename on a measured quantity means the EA looks for a file that does not exist, starts from era 0 and orphans a trained model, silently, because a missing cache is the normal first-run state. The shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the four derived fields rather than diffing them; a mismatch there would discard a fully-trained model over nothing the user did. Two fields appended to the .cfg for the conv/LSTM stages, length-guarded on read because FileReadInteger past EOF returns 0 with no error. ForceHiddenLayers, a compile-time constant like DebuggingMode, pins depth for diagnostic comparisons. It joins the fingerprint only when non-zero, so forced depths get their own files - sequential comparisons only, not simultaneous from one .ex5. Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64, 3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from ~58k to ~28k weights. Both builds compile 0 errors, 0 warnings. Re-keys existing models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
//--- m_initialNeuronsCount, so it MUST be called after ComputeFirstLayerWidth and before the
//--- fingerprint - see the note on fingerprint-feeding members at the top of this file.
int ComputeHiddenLayerCount(void) const;
fix: stop a .nnw from pinning a superseded architecture A .nnw persists the ARCHITECTURE, not just the weights: Save writes (int)activation per neuron and Load reads it straight back. The activation chosen in BuildFreshTopology() therefore only ever reached a brand-new topology - every reload restored the file's value and the next save wrote it back out, so a wrong value could never heal while the source read as though it were already fixed. That is how five models kept training with an unbounded NONE classification head for a full day after the 07-28 revert to SIGMOID. Confirmed by parsing the binaries: 848cb42c.nnw / 2e754b43.nnw carry `act=NONE` on the 3-neuron output layer, while a genuinely reset model of the same config carries act=SIGMOID. In the log it showed as negative "OOS raw out" values - impossible under sigmoid - escalating to a 4.14e13 logit spread with all three classes numerically identical (input-independent output) and balanced accuracy pinned on the 33.3% one-class floor. - OutputLayerActivation() is now the single source of truth, called by both BuildFreshTopology() and the new load-time repair, so the two can no longer diverge the way a duplicated literal did. - CNet::EnforceOutputActivation() re-asserts it after Load and reports the stale value; CExpertSignalAIBase::EnforceTopologyContract() logs the repair loudly, since weights learned under the old head may not be worth keeping even once the head is corrected. - Hidden layers are deliberately left alone: they legitimately differ per stage (PRELU dense/conv, NONE pool, TANH LSTM). Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 12:00:40 -04:00
//--- Re-assert everything about a just-loaded net that lives in the FILE but is owned by the CODE.
//--- Call after every successful Net.Load(); no-ops (and stays silent) when the file already agrees.
void EnforceTopologyContract(void);
//--- common network bootstrap: indicators, topology build/load, training-file bookkeeping
bool InitNeuralNetwork(CIndicators *indicators);
//--- The retrain-affecting configuration, as one string whose hash names the .nnw/.cfg pair.
//--- Body and the two rules that govern what may enter it: AIBase\Topology.mqh.
string BuildModelFingerprint(void);
//--- Exclusive per-config claim, so two charts can never train into one set of model files.
bool AcquireConfigLock(void);
void ReleaseConfigLock(void);
feat(chart): signal marks become price LEVELS at the trigger, not arrows beside the candle User request: 'move from arrows on lows and highs to small horizontal lines at the actual prices the entry/exit would trigger, just a bit larger than the candles. dark green for buy, dark red for sell.' Every mark is now an OBJ_TREND segment with both anchors at one price and both rays off, spanning 1.3 bar widths, drawn at the bar's CLOSE - the price a market order actually fires at, and the exact entry TripleBarrierLabel assumes. It used to sit on the candle's LOW for a Buy and its HIGH for a Sell: prices the trade never touches, picked so an arrow glyph would clear the candle. The tooltip now carries that price too. COLOUR NOW MEANS DIRECTION AND ONLY DIRECTION on every layer (dark green / dark red). Layer moves to width+style - the traded vote is solid and thick and drawn in front, a single model's raw opinion is thin, dotted and behind the candles - which keeps the distinction the old palette existed to draw (a model's opinion must never read as a trade) while freeing colour to say one thing consistently. Consequences handled, all of them the same 'a typed scan went blind' failure: - SaveChartSignals filtered OBJPROP_TYPE == OBJ_ARROW and read OBJPROP_ARROWCODE. It now filters OBJ_TREND and recovers direction from the colour. The sidecar keeps the old 217/218 numbers as its buy/sell token deliberately, so existing .arrows files still load. - AdvanceChartSignalRestore now rebuilds through the SAME creation point the live path uses, so a restored mark and a fresh one are identical objects. - The rescan-scoped delete enumerated ObjectsTotal(OBJ_ARROW) - retyped, or it silently deletes nothing. - ApplySignalsVisibility enumerated OBJ_ARROW with NO prefix filter. Under the new type that would have hidden and shown THE USER'S OWN trend lines on every Hide/Show click; it is now prefix-scoped. The old type was uncommon enough on a real chart to mask the missing check - trend lines are the most hand-drawn object there is. - DrawObject's high/low parameters are gone (6 call sites pass m_Close instead), so no caller can hand it a price it no longer draws at. - Fixed a pre-existing stale comment that still described the purge sweep as OBJ_ARROW-only three lines above the note explaining it had been widened to every type. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 14:11:49 -04:00
void DrawObject(datetime time, double signal, double close);
void DeleteObject(datetime time);
//--- Time-ordered NMS sweep over m_arrowSignalCache: prunes each same-direction run down to its
//--- earliest bar (deleting redundant neighbors within m_signalClusterWindow). Run once per era end.
void PruneDirectionalClusters(int bars);
//--- Whether BOTH directions can currently be traded, which is the precondition for the
//--- alternation rule in the NMS paths: with only one side enabled there is no opposite signal
//--- to wait for, so requiring alternation would suppress everything after the first call.
feat: 10-bar decluster window + alternation on every signal consumer SignalClusterWindow 3 -> 10 for all topologies. On H1 a 3-bar window collapsed only the tightest runs and left visible clusters at every turn; 10 bars is closer to the spacing of genuinely distinct setups. ALTERNATION. Rule 1 only collapses a same-direction run INSIDE the window; past it a second Buy is emitted with no Sell between, giving Buy/Buy/Buy/Sell. With both directions tradeable that sequence is the model re-entering a move it is already in rather than finding a new one. The kept sequence must now alternate: the first signal passes, and after that a direction passes only if the last KEPT signal was the opposite one. Added to ALL THREE consumers, with identical logic, because they must agree: - NmsLiveAccept -> the live trade - pass 3's OOS replay -> the tally the deploy gate grades - PruneDirectionalClusters -> the drawn history A rule applied to only some of these certifies one strategy and trades another - the same defect class as the geometry the gate certified while OpenParams placed something else (9a7c37f) - and would draw the user arrows the EA would never have taken. Deliberately NOT applied to the LABEL. The barrier target has no "must flip" invariant: consecutive Buy labels are routinely correct, and an earlier alternation gate was removed with the triple-barrier relabel for exactly that reason. This filters what is ACTED ON, which is what "applies to training" can honestly mean here - pass 3's declustered tally is the training-side number that decides deployment. BothDirectionsTradeable() is the stated precondition (with one side disabled there is no opposite to wait for, so alternation would suppress everything after the first call). This build has no long-only/short-only input, so it is constant true - kept as a named predicate so a future direction restriction has one place to change rather than three call sites silently assuming both sides. Build tag -> nms-alternate-v4. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 14:26:12 -04:00
bool BothDirectionsTradeable(void) const { return true; }
//--- Live newest-bar NMS accept test (time-keyed, idempotent per bar time - see m_signalClusterWindow).
bool NmsLiveAccept(datetime barTime, ENUM_SIGNAL dir, double conf)
{
if(m_signalClusterWindow <= 0)
return true;
if(dir != Buy && dir != Sell)
return true;
// Idempotent re-eval of the same bar (RefreshLatestSignal can run more than once per bar).
if(dir == Buy && m_nmsLiveBuyTime == barTime)
return m_nmsLiveBuyAccept;
if(dir == Sell && m_nmsLiveSellTime == barTime)
return m_nmsLiveSellAccept;
long minGap = (long)m_signalClusterWindow * PeriodSeconds();
datetime lastSame = (dir == Buy) ? m_nmsLiveBuyTime : m_nmsLiveSellTime;
bool accept;
// 1) Same-direction contiguous collapse: suppress if within the window of the previous SEEN
// same-direction bar (advance last-seen below either way, so a whole run collapses to one).
if(lastSame != 0 && (long)(barTime - lastSame) <= minGap)
accept = false;
else
{
// 2) Cross-direction resolution vs the last KEPT opposite signal: keep the stronger side.
accept = true;
if(m_nmsLiveKeptTime != 0 && m_nmsLiveKeptDir != dir &&
(long)(barTime - m_nmsLiveKeptTime) <= minGap)
{
if(conf > m_nmsLiveKeptConf)
DeleteObject(m_nmsLiveKeptTime); // this bar is stronger: remove the weaker opposite arrow
else
accept = false; // the kept opposite is stronger: suppress this bar
}
//--- 3) ALTERNATION. Rule 1 only collapses a same-direction run inside the window; past
//--- it, a second Buy is emitted with no Sell in between, giving Buy/Buy/Buy/Sell.
feat: 10-bar decluster window + alternation on every signal consumer SignalClusterWindow 3 -> 10 for all topologies. On H1 a 3-bar window collapsed only the tightest runs and left visible clusters at every turn; 10 bars is closer to the spacing of genuinely distinct setups. ALTERNATION. Rule 1 only collapses a same-direction run INSIDE the window; past it a second Buy is emitted with no Sell between, giving Buy/Buy/Buy/Sell. With both directions tradeable that sequence is the model re-entering a move it is already in rather than finding a new one. The kept sequence must now alternate: the first signal passes, and after that a direction passes only if the last KEPT signal was the opposite one. Added to ALL THREE consumers, with identical logic, because they must agree: - NmsLiveAccept -> the live trade - pass 3's OOS replay -> the tally the deploy gate grades - PruneDirectionalClusters -> the drawn history A rule applied to only some of these certifies one strategy and trades another - the same defect class as the geometry the gate certified while OpenParams placed something else (9a7c37f) - and would draw the user arrows the EA would never have taken. Deliberately NOT applied to the LABEL. The barrier target has no "must flip" invariant: consecutive Buy labels are routinely correct, and an earlier alternation gate was removed with the triple-barrier relabel for exactly that reason. This filters what is ACTED ON, which is what "applies to training" can honestly mean here - pass 3's declustered tally is the training-side number that decides deployment. BothDirectionsTradeable() is the stated precondition (with one side disabled there is no opposite to wait for, so alternation would suppress everything after the first call). This build has no long-only/short-only input, so it is constant true - kept as a named predicate so a future direction restriction has one place to change rather than three call sites silently assuming both sides. Build tag -> nms-alternate-v4. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 14:26:12 -04:00
if(accept && BothDirectionsTradeable() && m_nmsLiveKeptTime != 0 && m_nmsLiveKeptDir == dir)
accept = false;
}
if(dir == Buy)
{
m_nmsLiveBuyTime = barTime;
m_nmsLiveBuyAccept = accept;
}
else
{
m_nmsLiveSellTime = barTime;
m_nmsLiveSellAccept = accept;
}
if(accept)
{
m_nmsLiveKeptTime = barTime;
m_nmsLiveKeptDir = dir;
m_nmsLiveKeptConf = conf;
}
return accept;
}
fix(chart): arrows survived the EA that drew them - persist, then clear Reported: on deinit the panel and status label go, the signal arrows stay. Two independent causes, both fixed here. 1. It was partly deliberate. ShutdownChartCleanup carried a second behaviour selected by a `preserveChartArrows` flag derived from the deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the arrows were left on the chart on purpose, to avoid a reload flicker. That branch IS the reported symptom, an operator cannot tell it apart from a cleanup that failed, and it was outright wrong whenever the reload changed the config - REASON_PARAMETERS means exactly that, and the preserved arrows then belonged to a model the chart no longer runs, with nothing marking them stale. It is gone, along with the flag and m_purgeChartOnDestruct. One path now: persist, clear, restore on the next attach. 2. Whatever remains was unfalsifiable. PurgeChart was a single ObjectsDeleteAll(prefix) whose return value was discarded, with no caller ever looking at the chart again - so "the arrows are still there" and "the arrows were never there" produced identical evidence, which is why the report survived three sessions. It now verifies: after the bulk delete it walks the OBJ_ARROW-typed list (a handful of objects, not the whole chart), deletes any surviving WarSig_ by name, and says so. Costs one typed scan when the bulk delete works, which is the normal case; names the root cause when it does not. Every failure mode of SaveChartSignals was also silent - it returned void and had three bare early returns. It returns bool now, logs the open error with the filename, and the shutdown purge is CONDITIONAL on it: for a converged model the chart objects are the only copy of its signal history (nothing redraws them - the renderer runs per training era and a deployed model has none left), so a chart left littered because the disk write failed beats a clean chart bought by destroying the history. Either way the log now says which happened. Also states the user's rule once, where arrows come back rather than across InitNeuralNetwork's several exits: no weights loaded for this config => clear the sidecar and start visually clean. A fresh run must not inherit calls it never made, and the first save would otherwise adopt them (the sidecar is rebuilt by scanning the chart). Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
int PurgeChart(void);
ENUM_SIGNAL DoubleToSignal(double value);
//--- Shared status-label formatting for all three of Train()'s era passes (pass 1 sequential
//--- scan/ display, pass 2 shuffled backProp, pass 3 post-training OOS scoring) - see
//--- m_isTrainQueue's and m_isPass2Active's declaration comments for why the era loop is now
//--- three passes instead of one.
void UpdateTrainingStatusLabel(const string &progressLine, double neuron0, double neuron1, double neuron2, double signalValue, bool forceRefresh = false);
//--- Per-instance (NOT a function-local static - see CExpertSignalCustom::Direction()'s declaration
//--- comment for why that distinction matters for a method shared across PAI/CONV/LSTM instances)
//--- wall-clock throttle gate for UpdateTrainingStatusLabel()'s ChartRedraw().
uint m_lastStatusLabelUpdateTick;
//--- Last values passed to UpdateTrainingStatusLabel() - cached (updated on EVERY call, throttled
//--- or not) so the forced era-end refresh above has something real to redraw with instead of a
//--- stale/zeroed placeholder, since no "current bar" exists once an era's own three passes are done.
double m_lastDisplayNeuron0, m_lastDisplayNeuron1, m_lastDisplayNeuron2, m_lastDisplaySignal;
//--- Latest OOS Buy/Sell recall (-1 = n/a), cached like m_lastDisplayNeuron0 so the panel can show it
//--- on every call rather than only at era end. On-chart because blended accuracy is what a trader
//--- sees by default, and a model can look good on it purely by calling Neutral often.
int m_lastBuyRecallPct, m_lastSellRecallPct;
//--- Turns the head's 3 SIGMOID values into a softmax distribution in place and returns the
//--- signed dPrevSignal convention (+P(buy), -P(sell), exactly 0.0 for neutral). Max-subtracted
//--- before exp() for stability.
double ApplyClassificationSoftmax(void);
//--- Post-hoc logit adjustment / prior correction: reads the raw softmax probabilities
//--- ApplyClassificationSoftmax() just left in TempData[0..2] and returns the PRIOR-CORRECTED
//--- signed decision (same +P'(buy)/-P'(sell)/0-neutral convention).
double AdjustedSignalFromSoftmax(void);
feat(hud): per-member neuron lines + a vote label that moves as the nets learn Both 2026-08-19 reports were the same staleness: every source behind the label was an ERA artifact (live cache refills at pass-3 completion, the snapshot copies once per era, dPrevSignal is the frozen purge-band edge bar) - so the readout stepped at era cadence at best, stayed glued to one direction, and lagged the era counter. DisplayInference(): throttled (4s, 1s across an era boundary), SIDE-EFFECT-FREE forward of the current decision bar (window ending on bar 1, same question the live path asks) through the LEARNER net. Batch-norm running stats are bracketed frozen/RESTORED via the new CNet::GetBatchNormFrozen() + CNeuronBatchNormOCL::StatsFrozen() - restore, not unfreeze, because a display tick can land between pass-3 chunks whose whole scan holds them frozen. Writes nothing a trading or training path reads (dPrevSignal, NMS state, tallies, watermarks all untouched; RefreshLatestSignal is not reusable here precisely because it writes all of them). LSTM safe by construction: h/c zeroed per forward. ProspectiveVote() reads the fresh forward as its FIRST source; the era-artifact chain becomes the fallback (meta head, warm-up, window holes). DisplayHudLine(): the reference library's training label, per ensemble member - name, output activations (softmax probs or raw scalar), the decision, its weighted vote (the exact consensus numerator term), era, recent average error, "(trn)" while not vote-capable. Rendered under the vote line in RefreshVoteReadout BEFORE the live-vote defer (member lines are telemetry, not tradable readings), coloured by the member's own direction in muted tones - the vote line's strict green-only-when-it-would-trade rule is untouched. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 08:48:34 -04:00
//--- Throttled, side-effect-free forward of the CURRENT decision bar for the HUD - body and the
//--- full why in AIBase\Inference.mqh. True when m_dispProbs/m_dispSignal hold a usable read.
bool DisplayInference(void);
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//--- Margin between the winning class and its best rival, from the softmax already in TempData.
//--- Returns <0 when the winner is Neutral (not a directional call, so no operating point
//--- applies) or when the outputs are unreadable.
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
double DirectionalMargin(void);
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- Reset / accumulate / fit, in the order the calibration walk calls them. See
//--- DIR_CONF_THRESHOLD_BINS and DIR_CONF_CALIB_PCT_OF_IS.
feat: excursion-size head (Stage 1, measurement only) Direction is closed - normalised asymmetry fails on three instruments with a working positive control, and the classifier's own best-of-999 era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is a different question and RANGE clears at ~4x its null. Checked the denomination before building on that, since the source memo warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is predictable" is a claim about travel RELATIVE to current ATR, not a restatement of "ATR is autocorrelated". It is exactly the part a fixed multiple (stop 3.31*ATR, target 1.64*ATR) discards. A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches ladder rung k) upward and downward. Survival parameterisation rather than regressing the multiple, because it needs nothing new from CNet: sigmoid outputs and the per-neuron delta the `total != 3` branch already applies (a quantile head would need a linear activation and a pinball gradient in Network.mqh, Network.cl and the DirectML path, on a class four topologies share). Targets are free - m_ladderUpAt already records first-touch age per rung with 0 meaning never reached. Separate net, not extra outputs on the classifier: more outputs would change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push the count off 3 - the exact condition backProp uses to select the joint softmax gradient the 3-class head depends on. The classifier is bit-for-bit unaffected and this is removable without trace. STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the constant per-rung base rate - the baseline a fixed ATR multiple already assumes - with both predictors fitted IS and evaluated OOS, so neither gets a look at the test set. Positive skill justifies Stage 2 (drive SL/TP and sizing off ExcursionQuantile, which is defined and deliberately uncalled). Zero or negative means ATR already carries everything and Stage 2 must not be built. Trains only on primary occurrences: the replay queue oversamples for CLASS balance, and a direction-balanced sample is a biased SIZE sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
//--- EXCURSION-SIZE HEAD - see Expert\AIBase\Excursion.mqh. Predicts how FAR price travels, never
//--- which way; Stage 1 measures whether it beats a constant ATR multiple and places no orders.
bool ExcursionBuildTopology(CArrayObj &topology);
bool ExcursionEnsureHead(void);
bool ExcursionTargets(int idx);
void ExcursionTrainStep(int idx);
void ExcursionScoreStep(int idx);
feat: race the excursion head against a trailing-quantile incumbent Beating a frozen global constant is the weakest admissible bar for replacing a global constant. The honest incumbent is a rolling rung frequency: it adapts to the volatility regime - exactly what the head claims to predict - and needs no model, no 760 inputs and no training. Implemented as a ring of per-bar outcome bitmasks (32 rungs fit one ulong), sized horizon + EXCURSION_TRAIL_WINDOW. The newest `horizon` entries are held back UNRESOLVED: a bar's rung outcomes are only known one horizon later, so using them would be lookahead and would flatter the incumbent into an opponent the head could never fairly beat. Pass 3 walks oldest-to-newest, so "pushed more than horizon bars ago" is exactly "resolved by now". Each push is O(rungs), not O(window). The head's decision-rung Brier is pro-rated to the trailing estimate's coverage before the ratio, since the incumbent only scores bars where its window is warm. This line is worth reading on its own, independently of the head: if the trailing quantile beats the global constant, that is a cheap risk-control win available with no machine learning at all - and it is the same number either way, so the run answers both questions in one pass. The ring is deliberately NOT reset per era - it estimates the market, not the era, and re-warming 500 bars every era would leave the incumbent unusable over the first chunk of every scoring pass, handing the head a free win on exactly those bars. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:57:11 -04:00
void ExcursionTrailPush(void);
feat: excursion-size head (Stage 1, measurement only) Direction is closed - normalised asymmetry fails on three instruments with a working positive control, and the classifier's own best-of-999 era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is a different question and RANGE clears at ~4x its null. Checked the denomination before building on that, since the source memo warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is predictable" is a claim about travel RELATIVE to current ATR, not a restatement of "ATR is autocorrelated". It is exactly the part a fixed multiple (stop 3.31*ATR, target 1.64*ATR) discards. A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches ladder rung k) upward and downward. Survival parameterisation rather than regressing the multiple, because it needs nothing new from CNet: sigmoid outputs and the per-neuron delta the `total != 3` branch already applies (a quantile head would need a linear activation and a pinball gradient in Network.mqh, Network.cl and the DirectML path, on a class four topologies share). Targets are free - m_ladderUpAt already records first-touch age per rung with 0 meaning never reached. Separate net, not extra outputs on the classifier: more outputs would change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push the count off 3 - the exact condition backProp uses to select the joint softmax gradient the 3-class head depends on. The classifier is bit-for-bit unaffected and this is removable without trace. STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the constant per-rung base rate - the baseline a fixed ATR multiple already assumes - with both predictors fitted IS and evaluated OOS, so neither gets a look at the test set. Positive skill justifies Stage 2 (drive SL/TP and sizing off ExcursionQuantile, which is defined and deliberately uncalled). Zero or negative means ATR already carries everything and Stage 2 must not be built. Trains only on primary occurrences: the replay queue oversamples for CLASS balance, and a direction-balanced sample is a biased SIZE sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
void ExcursionResetEraScores(void);
double ExcursionQuantile(bool upward, double tau);
void ExcursionReport(void);
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
void ResetDirConfHistogram(void);
void AccumulateDirConfSample(double margin, bool wasCorrect, bool isPrimaryBar);
void FitDirConfThreshold(void);
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- CALIBRATION BAND BOUNDS, in pass-1 bar indices (0 = newest bar, so LARGER index = OLDER).
//--- The era's bars lay out, newest to oldest:
int CalibPurgeBars(void) const { return (int)MathMax(m_barrierHorizonBars, 1); }
int CalibLoIndex(int oosCutoff) const { return oosCutoff + CalibPurgeBars(); }
//--- Zero (an empty band) whenever the era is too short to carve one without eating the training set;
//--- callers must treat that as "no calibration this era" and leave the threshold where it is.
int CalibBandBars(int totalIter, int oosCutoff) const
{
int isSpan = totalIter - CalibLoIndex(oosCutoff) - CalibPurgeBars();
if(isSpan <= 0)
return 0;
return (int)(isSpan * (DIR_CONF_CALIB_PCT_OF_IS / 100.0));
}
int CalibHiIndex(int totalIter, int oosCutoff) const
{ return CalibLoIndex(oosCutoff) + CalibBandBars(totalIter, oosCutoff); }
//--- EMA-updates the persisted true class base rates (m_priorBuy/Sell/Neutral) from a just-finished
//--- era's true class counts. No-op on an empty/degenerate tally.
void UpdateClassPriors(long buyCnt, long sellCnt, long neutralCnt);
feat(ai): logit-adjusted loss, replacing oversampling and the post-hoc prior Menon et al. 2021 (ICLR), "Long-tail learning via logit adjustment": add tau*log(prior_c) to each class logit inside the training gradient. Softmax CE on adjusted logits is consistent for BALANCED error - the metric checkpoint selection already ranks on - so the loss and the deploy decision finally optimize the same thing. The engine already computed a true softmax + categorical-CE gradient and wrote it over the per-neuron sigmoid delta, so this is an offset added to three logits in the two places that gradient is built (backProp scalar path and backPropOCL). No backend, kernel or DLL change; the forward pass and every inference path are untouched, which is the point - the network learns to absorb the offset, so its raw argmax becomes the balanced-optimal decision with nothing applied at inference. Replaces rather than stacks. Minority replay is disabled while this is on, and the post-hoc inference prior is forced off. Stacking is not a theoretical worry: simulated on the measured 1118/1119/34298 distribution in the weak-signal regime, plain CE collapses to Neutral (33.4% balanced, Buy 0%), replay reaches 48.1%, logit adjustment 50.9% with better balance - and BOTH together score 45.4% with Neutral recall at 0%, worse than either alone. Buda et al. 2018 predicts exactly that. Motivation from the six-chart run: every topology took one direction to ~50% recall and abandoned the other, the direction chosen arbitrarily (the batch-norm control went Buy 1% / Sell 42%, the inverse of the other five). One era in 1,301 cleared the per-class recall floor. Fingerprinted conditionally, so the converged 60.7% models on disk keep their filenames and stay loadable as the fallback. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:05:14 -04:00
//--- Installs tau*log(prior_c) on Net from the freshly measured priors. Called once per era
//--- start, straight after UpdateClassPriors, so the offsets track the same distribution the
//--- era is scored against. No-op (and actively clears stale offsets) when the input is off.
void ApplyLogitAdjustment(void);
//--- Small binary sidecar (fileName + ".stats") persisting the calibration state that must survive a
//--- restart for live trading to behave like training: the true class priors and m_confidenceCalScale.
bool SaveModelStats(string fileName, bool common);
bool LoadModelStats(string fileName, bool common);
//--- Deploy-time (chart, backend present) self-check: runs the just-saved deployed model through
//--- both the backend and a temporary pure-MQL5 (CNet::SetCpuInference) clone on the same input
//--- window and returns true only if the outputs match within CPU_INFERENCE_MAX_DIFF.
bool ValidateCpuInference(void);
//--- Build the panel's "Buy/Sell accuracy: IS x% OOS y%" line (directional win-rate, Neutral excluded)
//--- from the cumulative counts (m_cumIsCorrect etc.); returns "...: measuring..." until at least one
//--- directional call has been validated. Shared by the training and live/complete simple panels.
string ComputeCompoundedAccuracyLine(void);
//--- Persist/restore the drawn directional arrows (the "WarSig_" objects) to a sidecar file so
//--- they survive an EA remove/re-add, recompile, or restart WITHOUT a retrain - the chart
//--- objects are destroyed on unload (destructor PurgeChart) and OnInit has no other way to
//--- bring them back.
fix(chart): arrows survived the EA that drew them - persist, then clear Reported: on deinit the panel and status label go, the signal arrows stay. Two independent causes, both fixed here. 1. It was partly deliberate. ShutdownChartCleanup carried a second behaviour selected by a `preserveChartArrows` flag derived from the deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the arrows were left on the chart on purpose, to avoid a reload flicker. That branch IS the reported symptom, an operator cannot tell it apart from a cleanup that failed, and it was outright wrong whenever the reload changed the config - REASON_PARAMETERS means exactly that, and the preserved arrows then belonged to a model the chart no longer runs, with nothing marking them stale. It is gone, along with the flag and m_purgeChartOnDestruct. One path now: persist, clear, restore on the next attach. 2. Whatever remains was unfalsifiable. PurgeChart was a single ObjectsDeleteAll(prefix) whose return value was discarded, with no caller ever looking at the chart again - so "the arrows are still there" and "the arrows were never there" produced identical evidence, which is why the report survived three sessions. It now verifies: after the bulk delete it walks the OBJ_ARROW-typed list (a handful of objects, not the whole chart), deletes any surviving WarSig_ by name, and says so. Costs one typed scan when the bulk delete works, which is the normal case; names the root cause when it does not. Every failure mode of SaveChartSignals was also silent - it returned void and had three bare early returns. It returns bool now, logs the open error with the filename, and the shutdown purge is CONDITIONAL on it: for a converged model the chart objects are the only copy of its signal history (nothing redraws them - the renderer runs per training era and a deployed model has none left), so a chart left littered because the disk write failed beats a clean chart bought by destroying the history. Either way the log now says which happened. Also states the user's rule once, where arrows come back rather than across InitNeuralNetwork's several exits: no weights loaded for this config => clear the sidecar and start visually clean. A fresh run must not inherit calls it never made, and the first save would otherwise adopt them (the sidecar is rebuilt by scanning the chart). Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
bool SaveChartSignals(bool pruneChartObjects = true);
void LoadChartSignals(void);
fix(chart): arrows survived the EA that drew them - persist, then clear Reported: on deinit the panel and status label go, the signal arrows stay. Two independent causes, both fixed here. 1. It was partly deliberate. ShutdownChartCleanup carried a second behaviour selected by a `preserveChartArrows` flag derived from the deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the arrows were left on the chart on purpose, to avoid a reload flicker. That branch IS the reported symptom, an operator cannot tell it apart from a cleanup that failed, and it was outright wrong whenever the reload changed the config - REASON_PARAMETERS means exactly that, and the preserved arrows then belonged to a model the chart no longer runs, with nothing marking them stale. It is gone, along with the flag and m_purgeChartOnDestruct. One path now: persist, clear, restore on the next attach. 2. Whatever remains was unfalsifiable. PurgeChart was a single ObjectsDeleteAll(prefix) whose return value was discarded, with no caller ever looking at the chart again - so "the arrows are still there" and "the arrows were never there" produced identical evidence, which is why the report survived three sessions. It now verifies: after the bulk delete it walks the OBJ_ARROW-typed list (a handful of objects, not the whole chart), deletes any surviving WarSig_ by name, and says so. Costs one typed scan when the bulk delete works, which is the normal case; names the root cause when it does not. Every failure mode of SaveChartSignals was also silent - it returned void and had three bare early returns. It returns bool now, logs the open error with the filename, and the shutdown purge is CONDITIONAL on it: for a converged model the chart objects are the only copy of its signal history (nothing redraws them - the renderer runs per training era and a deployed model has none left), so a chart left littered because the disk write failed beats a clean chart bought by destroying the history. Either way the log now says which happened. Also states the user's rule once, where arrows come back rather than across InitNeuralNetwork's several exits: no weights loaded for this config => clear the sidecar and start visually clean. A fresh run must not inherit calls it never made, and the first save would otherwise adopt them (the sidecar is rebuilt by scanning the chart). Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
//--- The shutdown half of that pair: persist, THEN clear the chart, and report both counts. See the
//--- definition for why the order is fixed and why the clear is conditional on the write.
void PersistAndClearChartSignals(void);
//--- How many arrows the last successful SaveChartSignals() wrote - reporting only.
int m_lastArrowsSaved;
//--- One-shot latch for PurgeChart()'s "saved N but the chart holds none" warning. PurgeChart
//--- runs twice on a clean removal - once from the shutdown path and again from the destructor,
//--- which is deliberate (the destructor covers teardowns that never reach OnDeinit) - and the
//--- second call necessarily finds an already-emptied chart with m_lastArrowsSaved still set.
fix: the Adam second moment was never Adam - all four tiers Root cause of the B=32 regression, and it predates F4 entirely. Every Adam kernel stored v already square-rooted and then fed that stored value back in as if it were the variance: v_new = sqrt(b2 * v_old + (1 - b2) * g^2) That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below unit scale, so the denominator stops tracking the gradient and Adam degrades into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll (batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where a scale-invariant optimizer gives the same distance for both. After the fix all six magnitudes read 1.199 and v tracks |g| exactly. It hit conv/LSTM specifically because they sit behind a batch-norm with running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep in the degraded regime - while the dense stack near the loss stayed in the working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/ 0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already squared v back for gamma/beta and its comment named the kernels as wrong, which is exactly why gamma/beta kept training while the stages behind froze. Persisted .nnw needs no migration - v keeps its std-dev meaning. Also, the two ways F4 exposed it, both mine: - No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods (Krizhevsky 2014; Granziol et al. 2022), applied once in InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD. - Plateau patience denominated in eras, so raising B made the ladder 32x more impatient in its only unit. PAI converged at era 41 on ~49k updates where the same config had been finding new bests at era 1028. TrainPlateauPatienceEras() stretches it by the same sqrt(B). TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23 eras per stage, not 8 -> 45). Both helpers are identities at B=1. Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into the ranking key rather than checked at deploy time so a one-sided era cannot become best-so-far in the first place. Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded CAppDialog teardown that sat ahead of it - the same ordering inversion the rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit (vs ~1.1 s for the three that finished) having reached none of its cleanup, so its arrows stayed on the chart. Steps are now timed in the log. PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now walks every object type and reports the object counts when both are zero. Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt. FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
bool m_purgeMismatchWarned;
fix: clear stale signal arrows when a fresh model starts at era 0 Arrow cleanup existed on two paths - the panel's reset-weights, and the topology-mismatch discard - but both are gated on there being a saved .nnw to delete. The third case had no cleanup at all: a fresh topology at era 0 with no weights behind it, which is what a changed config produces. A new fingerprint makes a new m_fileName, so the previous model's files are not "discarded", they are simply not this model's files, and nothing ever cleared the chart. That is not cosmetic. Arrows outlive the model that drew them twice over: 1. The chart objects live in the CHART, not the sidecar, so they survive a remove/re-add, a recompile, a restart and a fresh deploy no matter what happens to any file on disk. 2. SaveChartSignals() rebuilds the sidecar by SCANNING the chart for SIG_ARROW_PREFIX objects. So the first save of the fresh run adopts the dead model's calls and writes them out under the NEW model's filename - laundering them into the new model's history where nothing can separate them afterwards. Extracted the duplicated cleanup into ClearPersistedChartSignals(reason) - it cancels the deferred restore queue, deletes m_fileName + ".arrows", clears the namespaced chart objects and logs why - and called it from all three paths. The call sits at the BuildFreshTopology() call site, not inside it: the genetic tuner rebuilds a throwaway topology per candidate (AutoTune.mqh) and must never touch the chart. All three sites run after m_fileName has its config fingerprint appended, so they target the right sidecar. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 10:13:01 -04:00
//--- Wipe this model's drawn arrows AND their .arrows sidecar, plus any deferred restore still in
//--- flight. Call from every path that discards or replaces the trained weights - see the definition
//--- for why leaving them behind resurrects a dead model's calls through SaveChartSignals.
void ClearPersistedChartSignals(const string reason);
//--- Deferred ("async") half of LoadChartSignals: LoadChartSignals only PARSES the sidecar into
//--- the m_arrowRestore* buffers (an ~80KB read - instant) and returns, so OnInit never blocks;
//--- this then creates the chart objects in ARROW_RESTORE_BUDGET_MS slices, driven by the same
//--- 500ms timer that already paces training.
void AdvanceChartSignalRestore(void);
//--- parsed-but-not-yet-drawn arrows, consumed by AdvanceChartSignalRestore (see above)
datetime m_arrowRestoreTime[];
int m_arrowRestoreCode[];
double m_arrowRestorePrice[];
long m_arrowRestoreTf[];
int m_arrowRestoreIndex;
bool m_arrowRestorePending;
uint m_arrowRestoreStartMs;
//--- Deferred ("async") half of StartChartSignalRescan (public, defined inline further down):
//--- drains the per-bar inference loop in ARROW_RESTORE_BUDGET_MS slices off PollTraining's
//--- timer instead of blocking the button-click handler for however long a full lookback scan
//--- takes.
void AdvanceChartSignalRescan(void);
int m_rescanIndex;
int m_rescanHi;
int m_rescanBarsNow;
bool m_rescanPending;
uint m_rescanStartMs;
//--- Raw (PRE prior-correction) argmax tally, accumulated per-bar across AdvanceChartSignalRescan's
//--- slices - lets the completion log distinguish "the network itself calls Neutral almost everywhere"
//--- from "the network still discriminates, but AdjustedSignalFromSoftmax's logit-prior correction is
//--- suppressing it down to Neutral" - both produce an identical all-Neutral m_arrowSignalCache/empty
//--- chart otherwise.
int m_rescanRawBuy;
int m_rescanRawSell;
int m_rescanRawNeutral;
bool ResizeBuffers(int barIndex);
bool RefreshData();
research: export the feature matrix and a raw OHLCV grid for offline work The bottleneck on this project has never been the modelling - it is that every hypothesis costs a compile, a deploy, an attach and a log read, and answers exactly one question. Days have gone into questions that are seconds of arithmetic once the data is in hand. Adds a RESEARCH-ONLY build, gated behind WARRIOR_EXPORT_FEATURES and never compiled into a shipped binary, which writes two things to Common\Files\Warrior_EA\Research\ and then does nothing at all: <symbol>_<tf>_features.csv - one row per bar: index, time, OHLC, ATR, and the m_neuronsCount feature values. Exactly what the network sees. The raw bars ride along on purpose: with OHLC and ATR offline, every barrier geometry, horizon and in-trade target is recomputable without MetaTrader in the loop. <symbol>_<tf>_rates.csv - raw OHLCV across a grid of 8 symbols x 5 timeframes. The 26 engineered features only exist for the attached chart (indicator handles bind to PERIOD_CURRENT); raw rates do not, so ONE attach yields the whole research grid. The bar time also makes session/hour/day-of-week derivable - the only inputs in play that are not a transform of the same OHLCV series. Safety, because this binary gets attached to a chart on a LIVE ACCOUNT to reach real history: - OnTick returns immediately, so Expert.OnTick() - the entire trading path - is unreachable regardless of the AlgoTrading toggle, the signal state or the inputs. Structurally incapable of sending an order, not merely unlikely to. - No config lock. It never trains and never saves a model, so it has nothing to protect against a concurrent chart - and taking the lock would make it refuse to start exactly when the config it wants to read is already open, which is when it is most useful. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 15:49:57 -04:00
#ifdef WARRIOR_EXPORT_FEATURES
//--- RESEARCH BUILD ONLY, never compiled into a shipped binary. Dumps exactly what the network
//--- sees - one row per bar: index, time, OHLC, ATR, then the m_neuronsCount feature values - to
//--- a CSV under Common\Files\Warrior_EA\Research\.
research: export the feature matrix and a raw OHLCV grid for offline work The bottleneck on this project has never been the modelling - it is that every hypothesis costs a compile, a deploy, an attach and a log read, and answers exactly one question. Days have gone into questions that are seconds of arithmetic once the data is in hand. Adds a RESEARCH-ONLY build, gated behind WARRIOR_EXPORT_FEATURES and never compiled into a shipped binary, which writes two things to Common\Files\Warrior_EA\Research\ and then does nothing at all: <symbol>_<tf>_features.csv - one row per bar: index, time, OHLC, ATR, and the m_neuronsCount feature values. Exactly what the network sees. The raw bars ride along on purpose: with OHLC and ATR offline, every barrier geometry, horizon and in-trade target is recomputable without MetaTrader in the loop. <symbol>_<tf>_rates.csv - raw OHLCV across a grid of 8 symbols x 5 timeframes. The 26 engineered features only exist for the attached chart (indicator handles bind to PERIOD_CURRENT); raw rates do not, so ONE attach yields the whole research grid. The bar time also makes session/hour/day-of-week derivable - the only inputs in play that are not a transform of the same OHLCV series. Safety, because this binary gets attached to a chart on a LIVE ACCOUNT to reach real history: - OnTick returns immediately, so Expert.OnTick() - the entire trading path - is unreachable regardless of the AlgoTrading toggle, the signal state or the inputs. Structurally incapable of sending an order, not merely unlikely to. - No config lock. It never trains and never saves a model, so it has nothing to protect against a concurrent chart - and taking the lock would make it refuse to start exactly when the config it wants to read is already open, which is when it is most useful. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 15:49:57 -04:00
void ExportFeatureMatrix(void);
//--- Raw OHLCV for a grid of symbols/timeframes - see the definition for why the grid is worth more
//--- than the engineered features on their own.
void ExportRawRates(void);
#endif
bool BufferTempData(int idx);
//--- Assembles the full m_historyBars-wide input window ending AT bar r into TempData, OLDEST
//--- BAR FIRST. See the definition comment in AIBase\Features.mqh for the measurement behind
//--- that.
fix: the sequence models were reading the window backwards BuildFeatureWindow() replaces eight hand-rolled copies of the same loop and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first, because MQL5 timeseries indices run backwards and `r + b` with b ascending walks into the past. Harmless for PAI and CONV - a dense layer learns a weight per position either way, a conv learns time-mirrored kernels. Not harmless for the recurrent stacks: - LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t. - It writes output[] only when t == steps-1: the visible output IS the last hidden state. - c_t = f*c_{t-1} + i*g decays toward the start of the sequence. lstm_seq_flowcheck.cpp measured block 0's influence on the output at 1.2e-2 of block T-1's, at the shipped forget bias of 1.0. So the bar being PREDICTED sat at the far end of the decay and the output was handed to the OLDEST bar in the window - the exact inverse of what the window is for. ~80x backwards on LSTM and HYBRID, on all three tiers (OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never surfaced as a backend discrepancy. This does not create edge - the MI diagnostics read at the noise floor (p=0.4975) with a working positive control. It makes the one hypothesis those diagnostics explicitly do NOT cover testable: they are marginal and per-bar, and state they "cannot rule out one that only exists in combination or across time". The sequence model is the instrument for across-time structure and it has been crippled, so that hypothesis has never been honestly tested. Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and its features, so a stale .nnw would load cleanly and run a model fitted to one ordering against the other, silently. Re-keying every config is the point, not collateral damage. FORCES A FULL RETRAIN. Also: the now-relative bar caches are re-keyed on the two live paths. EnsureBarCachesCapacity() was only ever called from training paths, but once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar to RefreshConvergedSignal() and Train() is never re-entered - so nothing cleared the feature cache again for the life of the process. A chart that trained to convergence kept replaying the rows computed for the last training era's bar grid: the live signal froze at its convergence-time value, and OnlineLearnStep() backpropped those stale features against freshly resolved labels. Backtests were never affected (an inference-only process never allocates the arrays, so every read recomputes). Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 18:28:44 -04:00
bool BuildFeatureWindow(int r);
//--- shared by OnTickHandler() and the timer-driven PollTraining() - see definition
void ScheduleTrainingIfNeeded(void);
void Train(datetime StartTrainBar = 0);
fix: a restart no longer loses the measured geometry or the training window Terminal restart, 22:25: all four resumed models sat on empty windows with enum 2:6 barriers. Three interlocking causes, all visible in one log excerpt: 1) THE PRE-SCAN WINDOW WAS SIZED BY THE SAVED WATERMARK. A resumed model's dtStudied sits at its last studied bar, so Bars(dtStudied, now) ~ 0 and the resumed-model MI pre-scan built a zero-bar "complete" label cache - logged as "Buy: 0 | Sell: 0 | Neutral: 0". Train()'s own era start RESETS dtStudied to the training-window rule before computing its window; the pre-scan did not. The rule is now factored into TrainWindowStart() and both use it. The scan also refuses to arm before SERIES_SYNCHRONIZED (it ran in the same second as OnInit), and deployed models keep their watermark - for them it gates inference recency, not a training window. 2) THE HORIZON LATCHED ON AN INDICATOR WARM-UP. ComputeBarrierHorizonBars ran against a ZigZag with 0 calculated legs, fell back, and EnsureBarrierHorizon latched fallback(32) x slMult x tpMult = 384 for the process lifetime. A leg-starved horizon is now PROVISIONAL: re-resolved on the next rebuild, the label cache wiped if it moved (labels from two horizons answer different questions), and the geometry deriver refuses to run from it - a pair derived over a warm-up window would get PINNED. 3) THE DERIVED GEOMETRY WAS NEVER PERSISTED. The .cfg is written at model creation and at weights-reset - both BEFORE era 0 derives - so the measured pair lived only in memory: every restart read back zeros, adopted nothing, fell back to the enum barriers, and the era-0-only gate meant a resumed model could NEVER re-derive. A full day of training on 3.33/1.62 resumed as 2:6. Now: the settled pair is pinned to the .cfg the moment derivation completes (one-shot, atomic write), and the derive gate accepts any model with no pinned pair, not just era 0 - mid-run stability is carried by m_geometryDerived itself, which never allows a second derivation. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 22:40:43 -04:00
//--- the training window's start time - shared by Train()'s era start and the label-cache pre-scan
datetime TrainWindowStart(datetime startTrainBar);
//--- outer loop around Train(): when AutoTuneIndicators is on, tries randomized AD indicator
//--- input variations across m_indicatorTuneTrials calls to Train(), keeping the best-OOS one
void TuneIndicatorsAndTrain(datetime StartTrainBar = 0);
fix: live inference queried the 1-tick forming bar - a window training never built RefreshLatestSignal ran at the first tick after a bar opens and built its window at r=0: series index 0 at that instant is a candle with one tick of data - (close-open)/atr ~ 0, high ~ low, degenerate volume, indicators on a 1-tick bar. Training never produces such a window (every labeled bar is fully closed, entry at that bar's CLOSE), so the deployed model's final timestep - the one the LSTM/HYBRID output is keyed to - was out-of-distribution on every live decision, and pass 3's deploy-gate OOS scores measured a different query than live executed. The parity index is r=1: the newest CLOSED bar, whose close IS the current price - the exact instant the label's hypothetical entry happens. Single backtests shared the old skew (same r=0), which is why the tester agreed with live while both disagreed with training. Bookkeeping split that the index change forces: m_lastBarTime/dtStudied stay anchored to the FORMING bar's open (they gate against SERIES_LASTBAR_DATE; anchoring at bar 1 would re-fire the refresh every tick), while bt - the arrow, its High/Low placement, and NMS declustering - anchors to the decision bar, now matching the rescan path's convention. Also: a failed refresh no longer trades the previous bar's signal for the whole bar. RefreshLatestSignal returns success, zeroes dPrevSignal on failure (no opinion beats a stale one), and RefreshConvergedSignal advances dtStudied only on success so the next tick retries - the tester path (m_lastBarTime) already worked this way; this is the live path catching up. FORCES RE-VALIDATION of deployed models: the effective live query distribution changes. Bundled with the backprop transpose fix's retrain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 18:10:23 -04:00
//--- recomputes dPrevSignal/chart arrow for the newest CLOSED bar (bar 1 - see the definition's
//--- 2026-08-11 parity comment); used after restoring a checkpointed model at the end of Train()
//--- so the live signal matches the deployed weights.
fix: live inference queried the 1-tick forming bar - a window training never built RefreshLatestSignal ran at the first tick after a bar opens and built its window at r=0: series index 0 at that instant is a candle with one tick of data - (close-open)/atr ~ 0, high ~ low, degenerate volume, indicators on a 1-tick bar. Training never produces such a window (every labeled bar is fully closed, entry at that bar's CLOSE), so the deployed model's final timestep - the one the LSTM/HYBRID output is keyed to - was out-of-distribution on every live decision, and pass 3's deploy-gate OOS scores measured a different query than live executed. The parity index is r=1: the newest CLOSED bar, whose close IS the current price - the exact instant the label's hypothetical entry happens. Single backtests shared the old skew (same r=0), which is why the tester agreed with live while both disagreed with training. Bookkeeping split that the index change forces: m_lastBarTime/dtStudied stay anchored to the FORMING bar's open (they gate against SERIES_LASTBAR_DATE; anchoring at bar 1 would re-fire the refresh every tick), while bt - the arrow, its High/Low placement, and NMS declustering - anchors to the decision bar, now matching the rescan path's convention. Also: a failed refresh no longer trades the previous bar's signal for the whole bar. RefreshLatestSignal returns success, zeroes dPrevSignal on failure (no opinion beats a stale one), and RefreshConvergedSignal advances dtStudied only on success so the next tick retries - the tester path (m_lastBarTime) already worked this way; this is the live path catching up. FORCES RE-VALIDATION of deployed models: the effective live query distribution changes. Bundled with the backprop transpose fix's retrain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 18:10:23 -04:00
bool RefreshLatestSignal();
//--- inference-only "new bar" handler used once m_trainingComplete is true - see
//--- ScheduleTrainingIfNeeded()'s declaration comment for why this must NOT call Net.backProp()
void RefreshConvergedSignal(void);
//--- Online continual-learning step (live chart only) - see its implementation comment and the
//--- ONLINE_LEARN_* tunables. No-op in the tester/optimizer (m_inferenceOnly) and while training
//--- is active.
void OnlineLearnStep(void);
//--- Alpha-balanced focal sample weight (Lin et al. 2017 eq. 5) for ONE streamed bar - see the
//--- ONLINE_LEARN_* block's CLASS IMBALANCE comment for the derivation. Returns 1.0 for the
//--- regression head (no class structure).
double OnlineSampleWeight(ENUM_SIGNAL trueSignal, double pBuy, double pSell, double pNeutral);
//--- lazily bootstraps m_shadowNet if it's still NULL: tries loading a persisted shadow file
//--- first (continuity across EA restarts), falling back to cloning Net's current weights (via
//--- the same Save()/Load() pattern StartOosContinualSimulation() uses for m_simOosNet) if no
//--- compatible shadow file exists yet.
void EnsureShadowNet(void);
//--- persists m_shadowNet alongside every Net.Save() call, using the same run metadata (error/
//--- undefine/forecast/era/trainingComplete/indicator params) the caller already computed for
//--- Net.Save() itself - see m_shadowNet's declaration comment.
void SaveShadowNet(const double &indicatorParams[]);
//--- method of initialization of the indicators
bool InitOpen(CIndicators *indicators);
bool InitClose(CIndicators *indicators);
bool InitHigh(CIndicators *indicators);
bool InitLow(CIndicators *indicators);
bool InitVolumes(CIndicators *indicators);
bool InitTime(CIndicators *indicators);
//--- addToCollection=false is used by ReInitADIndicators() to rebuild an already-collected
//--- handle's params (here: a re-tuned period) without re-adding the (same) pointer into
//--- indicators a second time
bool InitMA(CIndicators *indicators, bool addToCollection = true);
bool InitRSI(CIndicators *indicators, bool addToCollection = true);
bool InitMACDFeature(CIndicators *indicators, bool addToCollection = true);
bool InitIchimoku(CIndicators *indicators, bool addToCollection = true);
bool InitADCumulativeDelta(CIndicators *indicators, bool addToCollection = true);
bool InitADShorteningOfThrust(CIndicators *indicators, bool addToCollection = true);
bool InitADWyckoffEventStream(CIndicators *indicators, bool addToCollection = true);
bool InitADWyckoffFailedStructure(CIndicators *indicators, bool addToCollection = true);
bool InitADWyckoffSignificantBarInversion(CIndicators *indicators, bool addToCollection = true);
bool InitADZigZag(CIndicators *indicators, bool addToCollection = true);
//--- common=false targets a LOCAL (non-shared) file - used by the tester/optimizer per-agent
//--- weight cache so cross-pass reuse never touches the production FILE_COMMON config/weights.
feat(nn): derive dense depth, train on all history, pin the shape in .cfg Completes the derived-topology work. Three inputs removed. AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five entries instead of eight. Depth is now derived from the two endpoints the taper already has to connect (derived first-layer width, output-tied final width) at a 2x per-layer compression target, clamped [2..5]. Asking a user to pick a layer count while the code derives the widths those layers taper between was asking for half a decision: at 64 units tapering to 12, four layers compress by 1.4x per step and five by 1.3x, so the extra depth bought no abstraction. On the shipping H1/10y default the derivation lands on 3 layers - the depth that actually won Run 2. StudyPeriods removed. There is no case for training on less data than the broker provides at a ~6% directional base rate; the honest generalization read comes from the OOS holdout, not from withholding history. Training now starts at the earliest available bar, floored by MinTrainYear, which answers a different question (excluding dubious pre-history) and stays. That required closing the hazard the old code documented: the capacity budget now MEASURES the symbol's real bar count, and a topology derived from a measurement would widen as history downloads. Both ends are now pinned. Every derived value left the weights-filename fingerprint - keying a filename on a measured quantity means the EA looks for a file that does not exist, starts from era 0 and orphans a trained model, silently, because a missing cache is the normal first-run state. The shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the four derived fields rather than diffing them; a mismatch there would discard a fully-trained model over nothing the user did. Two fields appended to the .cfg for the conv/LSTM stages, length-guarded on read because FileReadInteger past EOF returns 0 with no error. ForceHiddenLayers, a compile-time constant like DebuggingMode, pins depth for diagnostic comparisons. It joins the fingerprint only when non-zero, so forced depths get their own files - sequential comparisons only, not simultaneous from one .ex5. Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64, 3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from ~58k to ~28k weights. Both builds compile 0 errors, 0 warnings. Re-keys existing models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
bool SaveTopologyConfiguration(string fileName, int initialNeuronsCount, int hiddenLayersCount, double neuronsReduction, int minNeuronsCount, int optimizationAlgo, int historyBars, int outputNeuronsCount, int neuronsCount, int studyPeriod, int minTrainYear, bool isInitialized, int stopTrainWR, int fractalPeriods, int convFilterCount, int lstmHiddenSize, bool common = true);
//--- The four DERIVED shape fields are by REFERENCE and are ADOPTED from the .cfg, not compared
//--- against it.
bool LoadAndCompareTopologyConfiguration(string fileName, int &initialNeuronsCount, int &hiddenLayersCount, double neuronsReduction, int minNeuronsCount, int optimizationAlgo, int &historyBars, int outputNeuronsCount, int neuronsCount, int minTrainYear, bool isInitialized, int stopTrainWR, int fractalPeriods, int &convFilterCount, int &lstmHiddenSize, bool common = true);
//--- Retry helpers for the tester/opt seed-copy race: a live chart's own atomic Save() (write
//--- .savetmp, then FileMove() over the real file) can hold the source or destination file for a
//--- moment, and a concurrent FileCopy/FileOpen from a Strategy Tester agent reading the SAME
//--- production file can hit a transient Windows sharing violation in that narrow window.
bool CopyFileWithRetry(string srcFileName, string dstFileName);
bool CopySharedFile(string srcFileName, string dstFileName, bool quiet);
bool LoadNetWithRetry(double &indicatorParams[]);
//--- input data
bool m_useVolumes;
bool m_useTime;
bool m_useATR;
//--- Uses its own period (m_indicatorTuner.maPeriod), fed as ATR-normalized OHLC distance-from-MA
//--- (4 values, same convention as the base close-open/high-open/low-open features) plus the MA's
//--- own bar-over-bar change (1 value, ATR-normalized like every other price-domain feature here -
//--- not volume's previous-bar-ratio scheme, since a moving average lives in price units and
//--- already has ATR as its natural scale reference). See BufferTempDataCompute()'s m_useMA block
//--- for the exact 5 values. maPeriod starts equal to the Classic Signals PeriodMA input (see
//--- CADIndicatorTuner's constructor) but may diverge from it once AutoTuneIndicators searches a
//--- trial - the Classic Signals MA vote itself is untouched by that search, since it needs no
//--- training/warm-up and there is nothing for a tuning trial to validate it against.
bool m_useMA;
//--- RSI is already a 0-100 oscillator, so the only transform needed is /100 to match every
//--- other feature's roughly [-1,1]/[0,1] scale - no ATR or distance normalization applies.
bool m_useRSI;
//--- MACD as 3 ATR-normalized values (main line, signal line, histogram) - see
//--- BufferTempDataCompute()'s m_useMACD block. ATR-normalized rather than left raw because the
//--- MACD lines live in price units, exactly like the MA feature.
bool m_useMACD;
//--- Ichimoku as 8 values - see BufferTempDataCompute()'s m_useIchimoku block for each. The
//--- feature block applies the +Kijun offset and never calls ChinkouSpan();
//--- Signals\SignalIchimoku.mqh's class comment documents the buffer convention in full, and the
//--- same reasoning governs both.
bool m_useIchimoku;
//--- Nine normalized swing-context features: 5 confirmed-pivot values plus 4 recent-price-action
//--- ones (Donchian position at 20/50 bars, 20-bar return, 20-bar SMA extension) giving fresh
//--- context the >=100-bar-stale pivot anchor cannot. Reads the same m_ADZigZag the labels come
//--- from, and is never tuned for the same reason the label side is not.
bool m_useSwingContext;
//--- see System\NewsRelevance.mqh's declaration comment for what this feature actually encodes
//--- (event proximity + impact, not actual-vs-forecast deviation) and why the forward-looking half
//--- of it isn't lookahead bias.
bool m_useNews;
int m_newsFeatureWindowMinutes;
fix(signals): revive a dead MA model, and demote Sanyaku from state to event Two defects surfaced by research/test_classic.py, both verified fixed by re-running the transcription against 178k bars of EURUSD H1. CSignalMA model 1 could never fire. For any recursive average - and MA_TYPE_EMA is the shipped default - MA(i) = a*Close(i) + (1-a)*MA(i+1), so DiffMA(i) = a * (Close(i) - MA(i+1)) DiffCloseMA(i) = (1-a) * (Close(i) - MA(i+1)) are positive multiples of one quantity and always share a sign. Model 1 asks for a close BELOW a RISING average, which is precisely the combination that identity forbids: 0.000% of bars, either direction, any symbol. The MQL5 standard library this was ported from defaults to MODE_SMA, where the two are merely correlated - the bug arrived with the EMA default, not with the port. Reading the slope one bar back (DiffMAPrev) breaks the tie for every MA type while keeping the model's stated meaning. Now fires on 7.92% of bars. CSignalIchimoku model 11 fired on 27% of bars at weight 100. Sanyaku is three standing STATES conjoined with no transition term, so it held across long stretches - and being last in the if-chain at the top weight, the module's highest-conviction reading was also its most common one, overwriting all eight event models below it on a quarter of all bars. The old comment rejected an event form because "demanding all three flip on the same bar would fire almost never" - true, but that is not the alternative. Kouten is the TURN: the ALIGNMENT transitions, and only one role need change for it to. Testing !Sanyaku(idx+1) fires once per aligned stretch. Now 2.17%, in line with Kumo breakout (2.4%) and the strong TK cross (1.1%). DataReady() extended one bar deeper to cover the lookback. Neither pattern showed edge before or after; this is about the models meaning what they say and the vote not being dominated by a constant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:14:34 -04:00
//--- Cross-asset panel: the only feature block here whose inputs are NOT a transform of this
//--- symbol's own OHLCV series.
fix(signals): revive a dead MA model, and demote Sanyaku from state to event Two defects surfaced by research/test_classic.py, both verified fixed by re-running the transcription against 178k bars of EURUSD H1. CSignalMA model 1 could never fire. For any recursive average - and MA_TYPE_EMA is the shipped default - MA(i) = a*Close(i) + (1-a)*MA(i+1), so DiffMA(i) = a * (Close(i) - MA(i+1)) DiffCloseMA(i) = (1-a) * (Close(i) - MA(i+1)) are positive multiples of one quantity and always share a sign. Model 1 asks for a close BELOW a RISING average, which is precisely the combination that identity forbids: 0.000% of bars, either direction, any symbol. The MQL5 standard library this was ported from defaults to MODE_SMA, where the two are merely correlated - the bug arrived with the EMA default, not with the port. Reading the slope one bar back (DiffMAPrev) breaks the tie for every MA type while keeping the model's stated meaning. Now fires on 7.92% of bars. CSignalIchimoku model 11 fired on 27% of bars at weight 100. Sanyaku is three standing STATES conjoined with no transition term, so it held across long stretches - and being last in the if-chain at the top weight, the module's highest-conviction reading was also its most common one, overwriting all eight event models below it on a quarter of all bars. The old comment rejected an event form because "demanding all three flip on the same bar would fire almost never" - true, but that is not the alternative. Kouten is the TURN: the ALIGNMENT transitions, and only one role need change for it to. Testing !Sanyaku(idx+1) fires once per aligned stretch. Now 2.17%, in line with Kumo breakout (2.4%) and the strong TK cross (1.1%). DataReady() extended one bar deeper to cover the lookback. Neither pattern showed edge before or after; this is about the models meaning what they say and the vote not being dominated by a constant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:14:34 -04:00
bool m_useCrossAsset;
CCrossAssetPanel m_crossAsset;
bool BuildCrossAssetPanel(int bars);
//--- Train->serve parity for the panel (2026-08-11): the pair set is a MEASURED property of the
//--- terminal, so like the derived barrier pair it is pinned in the .cfg, not the filename hash
//--- (see BuildConfigFingerprint's XA note).
string m_crossAssetPairsPinned;
bool m_crossAssetCfgSaved;
//--- Alternative-data panel (2026-08-16): the second feature block whose inputs are not a
//--- transform of this symbol's own series, and the first whose inputs are not derivable from
//--- the terminal at all - COT positioning, the VIX complex, macro series, collected and
//--- publication-stamped by research/altdata, served as plain CSVs.
bool m_useAltData;
//--- EnableAltData input, distinct from m_useAltData: the input says the OPERATOR wants the
//--- block, m_useAltData says it is actually contributing features (input on AND file present
//--- AND >=1 column).
bool m_altDataEnabled;
//--- One-shot guard for the "data landed after the model was pinned" warning - the upkeep tick
//--- runs every 30 minutes and this must not become a recurring line nobody reads.
bool m_altDataLateWarned;
CAltDataPanel m_altData;
string m_altDataNamesPinned;
string ReadAltDataPinFromCfg(void);
//--- Spread as a feature. Measured as the strongest single feature in research/test_spread.py,
//--- though see the feature block for what it actually encodes and why that is less than it
//--- first appears.
feat(ai): spread as a volatility-regime feature, and fix a stale-index cache in both new blocks Adds spread/ATR and the spread change ratio as network inputs (EnableSpreadFeature, default on). Spread is the one microstructure channel that is both FX-available and genuinely historical in the Strategy Tester - "during testing, the spread is not modeled but is taken from historical data" - so unlike swap, signed tick flow or depth of market it is something a backtest can honestly validate. What it encodes, stated precisely because the raw measurement overstates it. research/test_spread.py found spr/atr the strongest single feature in this codebase, on 5 of 8 instrument/geometry cells at 2-4x any volume feature. But the barrier LABEL charges the spread inside its own barriers, so a wide-spread bar is mechanically likelier to resolve as a loss and the feature would partly be predicting its own cost model. Relabelling at zero cost and re-measuring the identical feature showed 20-40% of it WAS that tautology and the majority was not (XAUUSD retained 97%). What survives is a volatility-regime reading: spread is near-fixed while ATR is not, so the ratio runs high exactly when realised volatility is below its own ATR estimate, which genuinely predicts whether ATR-scaled barriers get reached. It is UNSIGNED - Neutral-vs-directional only, never a side. Also fixes a stale-index bug I introduced with the cross-asset panel and had just repeated in the spread series. Both cached on length alone: if(m_crossAsset.Bars() >= bars) return true; MQL5 series indices are relative to NOW, so one new closed candle shifts every index by one. Keyed only on length, the panel keeps serving its index 0 as a bar that is no longer the newest, and every cross-asset value is read one bar out of step with the price features sitting beside it in the same vector - silently, with no error and no shape change. This is the same class of defect as the dtStudied watermark behind the zero-direction backtests. Both now carry a datetime anchor on m_Time.GetData(0), the same invalidation key the label/feature bar caches already use. And a performance fix that fell out of it: with correct invalidation the panel rebuilds on every new bar, and RefreshConvergedSignal runs per bar - which in the tester would mean one full multi-symbol resample per simulated bar at training depth. Inference only reads bars 0..m_historyBars-1 plus the panel's own slow window, so it now requests exactly that. The cache check is >=, so a deeper panel left from training still satisfies it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:42:40 -04:00
bool m_useSpreadFeature;
int m_spreadSeries[];
int m_spreadSeriesBars;
//--- Newest bar the copy was anchored to. Same invalidation key the label/feature bar caches use
//--- (see EnsureBarCachesCapacity) and the same failure the zero-direction hunt traced.
feat(ai): spread as a volatility-regime feature, and fix a stale-index cache in both new blocks Adds spread/ATR and the spread change ratio as network inputs (EnableSpreadFeature, default on). Spread is the one microstructure channel that is both FX-available and genuinely historical in the Strategy Tester - "during testing, the spread is not modeled but is taken from historical data" - so unlike swap, signed tick flow or depth of market it is something a backtest can honestly validate. What it encodes, stated precisely because the raw measurement overstates it. research/test_spread.py found spr/atr the strongest single feature in this codebase, on 5 of 8 instrument/geometry cells at 2-4x any volume feature. But the barrier LABEL charges the spread inside its own barriers, so a wide-spread bar is mechanically likelier to resolve as a loss and the feature would partly be predicting its own cost model. Relabelling at zero cost and re-measuring the identical feature showed 20-40% of it WAS that tautology and the majority was not (XAUUSD retained 97%). What survives is a volatility-regime reading: spread is near-fixed while ATR is not, so the ratio runs high exactly when realised volatility is below its own ATR estimate, which genuinely predicts whether ATR-scaled barriers get reached. It is UNSIGNED - Neutral-vs-directional only, never a side. Also fixes a stale-index bug I introduced with the cross-asset panel and had just repeated in the spread series. Both cached on length alone: if(m_crossAsset.Bars() >= bars) return true; MQL5 series indices are relative to NOW, so one new closed candle shifts every index by one. Keyed only on length, the panel keeps serving its index 0 as a bar that is no longer the newest, and every cross-asset value is read one bar out of step with the price features sitting beside it in the same vector - silently, with no error and no shape change. This is the same class of defect as the dtStudied watermark behind the zero-direction backtests. Both now carry a datetime anchor on m_Time.GetData(0), the same invalidation key the label/feature bar caches already use. And a performance fix that fell out of it: with correct invalidation the panel rebuilds on every new bar, and RefreshConvergedSignal runs per bar - which in the tester would mean one full multi-symbol resample per simulated bar at training depth. Inference only reads bars 0..m_historyBars-1 plus the panel's own slow window, so it now requests exactly that. The cache check is >=, so a deeper panel left from training still satisfies it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:42:40 -04:00
datetime m_spreadSeriesAnchor;
datetime m_crossAssetAnchor;
bool EnsureSpreadSeries(int bars);
bool m_useADCumulativeDelta;
bool m_useADShorteningOfThrust;
bool m_useADWyckoffEventStream;
bool m_useADWyckoffFailedStructure;
bool m_useADWyckoffSignificantBarInversion;
public:
CExpertSignalAIBase(void);
~CExpertSignalAIBase(void);
//--- Reload the alt-data panel after CAltDataFetch rebuilt the feature CSV (OnTimer path, live
//--- only). Safe against the per-bar feature cache because new alt rows only ever matter to a
//--- NEW D1 bar, which resets that cache anyway.
void AltDataReload(void)
{
//--- Gated on the OPERATOR's switch, NOT on m_useAltData. m_useAltData latches false at init
//--- whenever the CSV was absent, so gating the reload on it made the EA structurally unable to
//--- consume data IT HAD JUST DOWNLOADED: on the first run after the alt-data folder is wiped -
//--- the normal pre-test routine here - the models are built ~30s BEFORE the fetch completes, the
//--- reload became a permanent no-op, and the entire run trained on price alone while a complete
//--- feature file sat on disk. Measured 2026-08-16 on SP500 H4: models pinned at fingerprint
//--- 6de8ba37 (0 alt features) at 19:36:40, SP500_D1.csv rebuilt with 13 features at 19:37:13,
//--- and every era after that trained without them - silently, because nothing looked again.
if(!m_altDataEnabled)
return;
int before = m_altData.FeatureCount();
m_altData.Load(m_symbol.Name(), (ENUM_TIMEFRAMES)m_period);
int after = m_altData.FeatureCount();
//--- Loading here is INERT while m_useAltData is false (both consumption sites gate on it),
//--- so this cannot widen the feature vector out from under a model whose width is already
//--- pinned.
if(before == 0 && after > 0 && !m_altDataLateWarned)
{
m_altDataLateWarned = true;
Print(ID + ": ALT DATA ARRIVED AFTER THIS MODEL WAS BUILT - " + IntegerToString(after) +
" features are on disk now, but this model's input width was pinned WITHOUT them, so it"
" is training on price alone and will keep doing so for the rest of this run."
" RE-ATTACH THE EA (or reload the chart) to build models that actually train on the"
" alt-data block. This is what happens when the alt-data folder is empty at attach time"
" and the EA downloads it moments later.");
}
}
//--- THE FRAMEWORK ENTRY POINT, and the only InitIndicators an AI signal needs: every subclass
//--- differs in TOPOLOGY (AddCustomLayers), never in how the net is brought up.
refactor(signals): AI signal files are identity + topology, nothing else Every AI signal repeated the same five-line InitIndicators override that did nothing but call InitNeuralNetwork. The cause was an access mismatch, not a design: CExpertSignalCustom declares InitIndicators public, the AI base redeclared it PROTECTED, and each subclass had to redeclare it public to be reachable by CExpert. Worse, the base's own override does a different job entirely - it creates the OHLC/ZigZag feature indicators - and InitNeuralNetwork called it back scope-qualified to stop the virtual dispatch landing in the subclass. Two jobs, one virtual name, and a recursion trap held off by a scope qualifier. The feature-indicator step is now InitFeatureIndicators() (protected, non-virtual, named for what it does) and the AI base carries the single public InitIndicators override. CONV/HYBRID/LSTM/PAI/META drop their copies and are now purely identity plus topology, which is the classic signal file's shape. Comment pass on ExpertSignalAIBase.mqh, -100 lines with every constant and every measured number kept. Three claims in the tier block were stale and inverted - it named CalibratedConfidenceMagnitude() as the tiering input where the code deliberately uses the RAW magnitude, and it described the signal DB as re-ranking each tier when ApplyPatternWeight declines the DB from the end of era 1. Also dropped a paragraph whose subject was a previous version of the comment, and moved two notes down onto the constants they document (CONV_COMPRESSION_DIVISOR was 16 lines and three unrelated defines away from its own text). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 08:57:54 -04:00
virtual bool InitIndicators(CIndicators *indicators) override
{
return InitNeuralNetwork(indicators);
}
feat(panel): commands reach signals down the filter tree, not through a registry The control panel drove training by looping g_aiSignals[] - a hand-maintained, MAX_AI_SIGNALS-capped, AI-only registry that had already dropped an ensemble member on the floor once (609be10). A model missing from it still trains and still votes, it just cannot be paused, stopped, deployed or reset, and every button label is computed from the same short list, so the panel described one set of models while acting on another. Classic signals could not respond to a panel action at all. Commands now walk the signal tree CExpert already owns: Expert.DispatchSignalCommand(cmd) -> root signal -> every filter, recursively, returning how many actually acted. CExpertSignalCustom carries the seam (OnSignalCommand / HasSignalTrait, both no-ops by default), so a classic signal opts in by overriding two methods and needs no registration and no cap. CExpertSignalAIBase implements the training commands over its existing Pause/Stop/Deploy/ Reset methods - the behaviour is unchanged, only its reach reported. Button labels ask the same tree via CountSignalTrait, with SIGTRAIT_TRAINABLE as an explicit denominator: "all paused" is meaningless without knowing how many could be paused. Pause/Stop resolve their toggle direction ONCE in the EA and hand every model the same plain command, instead of each re-deriving the direction from its own local state - which is how a mixed set ends up half paused. The alerts now report the count acted on rather than assuming it. Two dispatch bugs found on the way, both from a database guard copied onto event delivery: CExpertSignalCustom::OnTickHandler and ::OnChartEventHandler each skipped any filter whose GetFilterID() is "NULL". That id is a DB folder name, and CSignalNewsFilter, CSignalSessionFilter and CSignalRiskGuard never set one - so all three were silently receiving neither ticks nor chart events. The guard stays where it belongs, on the paths that write pattern tables. ENUM_CP_ACTION moves to Enumerations\GlobalEnums.mqh (now include- guarded) because the Expert bases have to name it and the panel is included long after them. The AI-only lifecycle loops - PollTraining, the weight autosave, AltDataReload, OnDeinit's shutdown cascade - still use g_aiSignals[] and are untouched here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 09:05:50 -04:00
//--- CONTROL PANEL. Every training action the panel offers arrives here, through the filter tree
//--- rather than through a registry - see CExpertSignalCustom::OnSignalCommand.
virtual bool OnSignalCommand(const ENUM_SIGNAL_COMMAND cmd) override;
virtual bool HasSignalTrait(const ENUM_SIGNAL_TRAIT trait) override;
//--- "voting" that price will grow/fall, common to every AI signal (single market model)
virtual int LongCondition(void);
virtual int ShortCondition(void);
//--- |dPrevSignal| is already a 0..1 confidence for classification output (softmax probability
//--- of the winning class) and typically bounded for regression output (tanh-activated network);
//--- OpenParams() clamps regardless.
double CalibratedConfidenceMagnitude(void) const
{
double mag = MathAbs(dPrevSignal);
if(!MathIsValidNumber(mag))
return 0.0;
if(m_outputNeuronsCount == 3)
mag = MathMin(1.0, mag * m_confidenceCalScale);
if(!MathIsValidNumber(mag))
return 0.0;
return mag;
}
virtual double AIConfidence(void) override { return CalibratedConfidenceMagnitude(); }
// Signed for direction-aware use (AI-driven early exit): sign matches dPrevSignal's
// convention (+ buy, - sell, 0 neutral/no signal yet). dPrevSignal == -2 is the
// "not yet studied" sentinel, not a real sell signal - treat it as no confidence.
virtual double SignedAIConfidence(void) override
{
if(dPrevSignal == -2)
return 0.0;
double sign = (dPrevSignal > 0.0) ? 1.0 : (dPrevSignal < 0.0) ? -1.0 : 0.0;
if(sign == 0.0)
return 0.0;
return sign * CalibratedConfidenceMagnitude();
}
//--- event handlers, common to every AI signal
virtual void OnTickHandler(void);
//--- drives the same training-scheduling check as OnTickHandler(), but callable from a timer so
//--- it isn't dependent on ticks (which don't arrive while the market is closed)
void PollTraining(void);
virtual void OnChartEventHandler(const int id,
const long &lparam,
const double &dparam,
const string &sparam);
//--- methods of adjusting "weights" of the 4 confidence-tier market models - see m_pattern_0's
//--- declaration comment
void Pattern_0(int value) { m_pattern_0 = value; }
void Pattern_1(int value) { m_pattern_1 = value; }
void Pattern_2(int value) { m_pattern_2 = value; }
void Pattern_3(int value) { m_pattern_3 = value; }
virtual void ApplyPatternWeight(int patternNumber, int weight);
feat(rank): AI models rank their own confidence tiers from held-out outcomes Closes the caveat 4858507 shipped with: the vote is a confidence percentage, but only to the extent the pattern weights are measured. AI tier weights sat at their designed defaults (25/50/75/100) because AI rows only ever arrive from LIVE journaling, of which a training run produces almost none. AND A STALE-TIER BUG THAT MADE THE EVIDENCE MEANINGLESS. The OOS scan bucketed every scanned bar by ConfidenceTier(), which reads dPrevSignal - and dPrevSignal is assigned in PASS 1 only, never anywhere in the OOS scan. So an entire era's fires were bucketed by one stale, unrelated bar's confidence and landed in a SINGLE tier. That is the "tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0)" symptom recorded on 2026-08-16 and attributed to the calibration clamp. The clamp was real and was fixed then; this is a second, independent cause of the identical output that survived that fix untouched - which is why the log kept reading the same afterwards. Two causes, one symptom. Now ConfidenceTierFor(adjSig): the bar this iteration actually scored. WHY THIS DOES NOT WRITE ROWS TO THE SIGNAL DB, which was the obvious reading of "fill the database during training". The user's own observation is the reason: a classic Pattern_2 is a fixed geometric condition, so its win rate is legitimately accumulated over years, but an AI Pattern_2 means "confidence landed in tier 2" and tier 2 under era 100's weights is a different statement from tier 2 under era 500's. The DB's value is ACCUMULATION, and accumulation is exactly what is wrong here - it would average together models that no longer exist, while colliding with the per-table row cap and mixing measured-on-holdout outcomes into the live ledger's own tables. What the DB actually supplies is a measured win rate per pattern, and pass 3 already computes that on held-out bars, thousands at a time. So the model ranks itself once per era, REPLACING rather than accumulating, which makes the weights describe the current weights by construction. ESTIMATOR. Not WinRateFromCounts(): it returns NO_DATA below 100 raw trades BEFORE shrinking, which here would fire on every tier every era and hand all four the pooled rate - the tiers could never separate and the mechanism would be inert. Shrinkage is the answer to a small sample; a floor in front of it means the shrinkage never runs. Instead: a Beta prior of TIER_PRIOR_EFF_N pseudo-observations centred on the model's pooled holdout rate, counted in EFFECTIVE observations, because overlapping triple-barrier labels mean 800 raw fires can be worth ~12 independent ones. Rounded to the integer, not to the decade NormalizeWinRate() uses, which would collapse the shrunk tiers back into one number. NO SAME-ERA CIRCULARITY, and it falls out of the ordering rather than a guard: weights are computed at the END of era N, so the vote scored during era N was cast with era N-1's weights. The deploy gate never grades a vote whose weights were fitted on the bars it is scoring. Residual leakage remains - the same OOS bars each era under a different model - and is stated in the code rather than papered over. Both DB clobber paths are closed: ApplyPatternWeight() declines once self-ranked, and UpdateSignalsWeights()' filter.Weight() call is guarded by SelfRanked() - guarding only the tiers would have let the hourly ranking pass undo half the self-ranking. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 16:00:32 -04:00
//--- Re-derives the four tier weights (and the module weight) from THIS era's held-out outcomes.
//--- Called once per era at the end of pass 3, when m_oosTierFired/Hits are complete.
void RankTiersFromOos(void);
feat(baselines): Alglib forest + linear on the NN's own matrix Every direction verdict so far was measured through one architecture family, so "flat" has two readings that no topology tuning can separate: the net is the wrong learner, or the matrix carries no directional information. Two learners with completely different inductive biases - Alglib's random decision forest and an ordinary least-squares fit - now train on the SAME feature windows (BuildFeatureWindow, the net's own function, so there is no second feature implementation to drift), the SAME labels, the SAME IS/OOS split with both purges, and are scored through the SAME precision-against-always-one-direction comparison and the same Sidak family-wise arithmetic the deploy gate uses. If both also land at chance, the matrix is the limit. Deliberate choices, each of which could have made the comparison a different question wearing this one's name: - LRBuild, not LRBuildZ: the intercept absorbs the class imbalance, and a baseline handicapped by a forced zero intercept would flatter the net for the wrong reason. - Raw call counts in the SE, matching the live gate's known-permissive test rather than correcting it here - both sides must face the same bar. - No threshold sweep on the linear fit: a threshold fitted on the slice being scored is the calibration leak the purged band exists to avoid. - Uniform stride when a cap bites, not the newest N rows, so a score difference cannot be a regime difference. What was dropped is logged. Ships off (Run_Alglib_Baselines = false): it is a measurement, not a trading feature, nothing trades on the answer and no model is saved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 23:45:03 -04:00
//--- NON-NN BASELINES on this model's own feature/label matrix - bodies and the full argument in
//--- AIBase\Baselines.mqh. One shot per run at a pass-3 completion, off unless the trader asks.
void RunBaselineComparison(const int bars, const int totalIter, const int oosCutoff);
int BaselineCandidateBars(const int lo, const int hi, const int bars,
const int cap, int &rows[]);
bool BaselineRowFeatures(const int bar, const int width, double &x[]);
fix(baselines): the diagnostic held the EA thread long enough for MetaTrader to kill the process Run_Alglib_Baselines was described as bounded because it 'cannot outrun the bar it runs on'. That bound is four hours on H4, it was never enforced, and it is the wrong target: the EA is single threaded, so while this pass runs there is no training, no panel, no tick handling, and a removal request only queues. Measured today on SP500 H4, 800 inputs x 4000 rows: forest 5 s, MLP 52 s, then the 3-fold CV - three more full retrains. At 15:18:16, 83 s into it, MetaTrader force-terminated the EA. OnDeinit never ran, so every arrow, panel object and label was orphaned on the chart. The init purge cleans that up on re-attach; nothing in the pass writes to a cache, saves a model or feeds a decision, so the damage was the freeze and the dirty chart, not corrupted state. Three changes: - BASELINE_MAX_CELLS. Every fit costs O(rows x width) and the row caps carried no width term. Rows now shrink as the window widens - 1000 instead of 4000 at 800 columns - and the log says when the cap bit. - BASELINE_BUDGET_MS with BaselineBudgetSpent(), replacing the bare ShutdownRequested() calls at each phase boundary. A stop and a spent budget need the same answer and only one was asked. - The cross-validation is now declined BEFORE it starts, from its predicted cost - the measured MLP time times the fold count. A boundary check cannot help once a phase is in flight, which is exactly the phase that was in flight when the process died. Also: the MLP now reports its weight count against its row count. At 800 inputs it carries ~6,400 weights, and today's run duly printed a training class error of 0.000. That is knowable from the shape before fitting, so it is stated rather than left to a cross-validation that may not run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 15:22:42 -04:00
//--- True when this pass must stop before `nextPhase` - stop requested, or the wall clock spent.
bool BaselineBudgetSpent(const uint startTick, const string nextPhase);
feat(baselines): Alglib forest + linear on the NN's own matrix Every direction verdict so far was measured through one architecture family, so "flat" has two readings that no topology tuning can separate: the net is the wrong learner, or the matrix carries no directional information. Two learners with completely different inductive biases - Alglib's random decision forest and an ordinary least-squares fit - now train on the SAME feature windows (BuildFeatureWindow, the net's own function, so there is no second feature implementation to drift), the SAME labels, the SAME IS/OOS split with both purges, and are scored through the SAME precision-against-always-one-direction comparison and the same Sidak family-wise arithmetic the deploy gate uses. If both also land at chance, the matrix is the limit. Deliberate choices, each of which could have made the comparison a different question wearing this one's name: - LRBuild, not LRBuildZ: the intercept absorbs the class imbalance, and a baseline handicapped by a forced zero intercept would flatter the net for the wrong reason. - Raw call counts in the SE, matching the live gate's known-permissive test rather than correcting it here - both sides must face the same bar. - No threshold sweep on the linear fit: a threshold fitted on the slice being scored is the calibration leak the purged band exists to avoid. - Uniform stride when a cap bites, not the newest N rows, so a score difference cannot be a regime difference. What was dropped is logged. Ships off (Run_Alglib_Baselines = false): it is a measurement, not a trading feature, nothing trades on the answer and no model is saved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 23:45:03 -04:00
void ReportBaselineModel(const string label, const int calls, const int hits,
const int scored, const double chancePct);
feat(baselines): matrix redundancy and a declustered regime test Two ALGLIB diagnostics on the pass that already builds the matrix, under the same Run_Alglib_Baselines switch, so they cost nothing by default and share one walk of the feature pipeline. REDUNDANCY (CBaseStat::PearsonCorrM + CPCAnalysis::PCABuildBasis). The alt-data block already produced this finding once, by hand: 13 features x 16 window slots = 208 inputs spanning ~11.5 effective dimensions, and the cost was not wasted parameters but a gradient weighting bias, since batch norm rescales collinear copies without decorrelating them. This generalises that measurement to every column instead of the one block someone thought to check. Run on the ANCHOR BAR'S ROW, not the full window. The window is m_historyBars near-copies of the same columns at different lags, so per-bar redundancy is the question worth asking - the lag structure is what the conv/LSTM stage exists to exploit - and it keeps the eigensolve at ~60x60 rather than ~960x960. REGIME STABILITY (CMannWhitneyU::CMannWhitneyUTest) on the net's own signed margin: its conviction, signed by whether the conviction paid, older half of the OOS window against newer. DECLUSTERED FIRST, which is the whole reason this is not a one-line call. Triple-barrier labels overlap, so consecutive scored bars share the price action that resolved them and are not independent draws. Feeding all of them to a rank test yields a p computed on an n it does not have - the same error label overlap invalidated everywhere else here. One sample per label horizon costs most of the rows and buys a p that means something; when too few survive, it says so and declines. Both report and neither acts. Wiring the regime flag to position sizing is the obvious next step and the one to take only after watching the number: run every era on ~60 effective observations, a 0.05 test crosses by chance regularly, and an auto-derisk on that is a random position size generator. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 00:01:34 -04:00
void ReportFeatureRedundancy(CMatrixDouble &rows, const int n);
void ReportRegimeStability(double &margins[], int &marginBars[]);
feat(baselines): geometry-drift check on the derived stop DeriveBarrierGeometry() reads the stop off a quantile of the adverse excursions in the IS region ONLY - correctly, since a geometry chosen with the holdout in view has used the holdout for selection and it stops being a holdout. The cost of that correct choice is that nothing ever checked whether the distribution it measured still holds on the bars the model actually trades. If adverse excursions run wider in the OOS window than in the IS region, the derived stop is too tight for the market it is used in, every label was cut on the wrong geometry, and the deploy gate certified a game the trade is not playing - the 2026-08-09 geometry mismatch arriving through drift rather than through a config error. Reported in TWO currencies deliberately. A rank-test p says whether the distributions differ; it does not say whether anyone should care. The stop each half's own quantile would derive says exactly that, in ATR multiples - the units the order is placed in. A significant p with both stops on the same ladder rung is a curiosity; half an ATR of movement is a problem whether or not it clears 0.05. Declustered first, same as the regime test: overlapping labels are not independent draws. Harvest guards are copied from the deriver's own so the two describe the same sample - and the split is verified identical (totalIter == bars - historyBars, so Train's oosCutoff and the deriver's are the same number). Runs before the two model fits and needs only the excursion caches: a chart too thin to fit a forest can still have drifted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 00:34:55 -04:00
void ReportGeometryDrift(const int bars, const int oosCutoff);
int CollectAdverseExcursions(const int lo, const int hi, double &out[]);
feat(baselines): combining-weight fit, MLP cross-validation, all-lags correlation Three ALGLIB additions, all measurement-only and all under the existing Run_Alglib_Baselines switch. MinBLEIC COMBINING WEIGHTS. The live ensemble weights each member by its own pooled holdout win rate - a defensible prior, but not a fit, and nothing has ever asked what mixture minimises error on the bars the members disagreed about. Two individually-mediocre members wrong in different places can beat one individually better, and a per-member win rate cannot express that because it never looks at them jointly. Solved on the simplex (w >= 0, sum w = 1), which is exactly what MinBLEIC is for. Non-negative because a negative weight asserts "trade the opposite of this member", a claim ~60 effective observations cannot support. Least squares on the signed outcome rather than precision: precision is a STEP function of the threshold that no gradient method can walk, and optimising a smooth proxy for a step decision is how c3daded put every operating point 14pp underwater - so the result is reported in BOTH currencies, the SSE it minimised and the directional hit rate the mixture would actually have scored against the equal mix. If the second does not improve, the first is noise. This needed data that did not exist: g_ensVoteSum accumulates member contributions and the sum destroys the decomposition, while g_ensVoteVoterMask records only WHETHER a member voted, never what. g_ensVoteMember[] keeps them unsummed. The live arithmetic is untouched. ENS_MAX_MEMBERS is 8 and deliberately larger than MAX_AI_SIGNALS (5): independent caps, over-allocating is free, and matching them would make this array silently short the day the registry grows - a cap that has already dropped a member once without saying so. MLPKFoldCVLBFGS. Every baseline row carries a binomial SE, which is the sampling error of SCORING a fixed model and says nothing about how much the FIT moves. One LBFGS run from one random start can land anywhere, and a baseline that cleared or missed the bar on luck of initialisation reads exactly like one that did it on merit. 3 folds, because each is a full retrain. LBFGS not LM - LM builds a Hessian over ~7,700 weights. CCorr ALL-LAGS PROFILE. Added BESIDE the MI lag profile, not instead: MI catches nonlinear dependence and is the stronger negative, which is why it settled the verdict - but its per-lag permutation null limits it to ~20 lags. FFT correlation gets every lag in one O(n log n) pass, so linear structure parked at lag 300 would surface for free. Different question, not a replacement. Walks CONTIGUOUS bars, unlike everything else in this file, because a lag index is meaningless otherwise; both series are mean-centred first since CorrR1D is a raw sum of products; and the max over columns x lags is judged against a Sidak family of exactly that size, not a bare 2-sigma line. fasttransforms.mqh needed its own include - verified that none of ap/optimization/statistics/solvers/linalg reaches it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 07:44:36 -04:00
void ReportLinearLagProfile(const int bars, const int oosCutoff);
void ReportCombiningWeights(void);
double CombinerSSE(const double &w[], const int members, const double &s[],
const double &t[], const int rows);
feat(consistency): the five review flaws fixed - training wears the live constraints, the gate wears the policy 1. Labels and the exit simulator go through the broker's stop-distance check: risk/reward widen to SYMBOL_TRADE_STOPS_LEVEL exactly as TCAdjustStops does at order time - the M5/tight-ATR case where live trades ran wider geometry than training measured. Current stops level stands in for history (like the spread); measured quantity, so it does not key the fingerprint. 2. The Intelligent drift verdict moved into RefreshDriftVerdict(), which RESCANS the label cache and now runs at every era end beside RankTiersFromOos - era-cadence instead of waiting for rare full rebuilds. Prints only on change. 3. Session filter is any-broker: sessions defined on their financial centres' civil clocks (London 08-16 Europe/London, NY 08-17 America/New_York, Tokyo 09-18 Asia/Tokyo), converted to UTC by each centre's own computed DST rule (EU last-Sun-Mar/Oct, US 2nd-Sun-Mar/1st-Sun-Nov), then to broker time by the MEASURED server-vs-GMT offset (half-hour brokers included). Windows may wrap midnight in broker time - the interval test handles it. Replaces the EET-hardcoded anchors, which were correct on exactly one broker and got Tokyo wrong by an hour each European summer. 4. The current-session-table-for-history caveat resolved by analysis: the bars bound the error - a too-late assumed close meets no bars (zero error), a too-early one truncates conservatively (<=1h, never optimistic, cannot manufacture edge). Documented at the site. 5. The ensemble deploy gate mirrors the direction policy: blocked-side fires are not fired bars (certified == traded), the zero-skill reference uses only ACHIEVABLE baselines (always-short is not a strategy a long-only book can run), and one-sidedness BY POLICY is not degeneracy - the two-sided requirement applies only when both sides are allowed. Sell predictions keep their other jobs (exit triggers, consensus dilution) untouched. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 12:14:01 -04:00
//--- The Intelligent-direction drift verdict, rescanned from the label cache - body and the
//--- full statistics note in AIBase\Labels.mqh. Runs at prebuild and at every era end.
void RefreshDriftVerdict(void);
feat(rank): AI models rank their own confidence tiers from held-out outcomes Closes the caveat 4858507 shipped with: the vote is a confidence percentage, but only to the extent the pattern weights are measured. AI tier weights sat at their designed defaults (25/50/75/100) because AI rows only ever arrive from LIVE journaling, of which a training run produces almost none. AND A STALE-TIER BUG THAT MADE THE EVIDENCE MEANINGLESS. The OOS scan bucketed every scanned bar by ConfidenceTier(), which reads dPrevSignal - and dPrevSignal is assigned in PASS 1 only, never anywhere in the OOS scan. So an entire era's fires were bucketed by one stale, unrelated bar's confidence and landed in a SINGLE tier. That is the "tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0)" symptom recorded on 2026-08-16 and attributed to the calibration clamp. The clamp was real and was fixed then; this is a second, independent cause of the identical output that survived that fix untouched - which is why the log kept reading the same afterwards. Two causes, one symptom. Now ConfidenceTierFor(adjSig): the bar this iteration actually scored. WHY THIS DOES NOT WRITE ROWS TO THE SIGNAL DB, which was the obvious reading of "fill the database during training". The user's own observation is the reason: a classic Pattern_2 is a fixed geometric condition, so its win rate is legitimately accumulated over years, but an AI Pattern_2 means "confidence landed in tier 2" and tier 2 under era 100's weights is a different statement from tier 2 under era 500's. The DB's value is ACCUMULATION, and accumulation is exactly what is wrong here - it would average together models that no longer exist, while colliding with the per-table row cap and mixing measured-on-holdout outcomes into the live ledger's own tables. What the DB actually supplies is a measured win rate per pattern, and pass 3 already computes that on held-out bars, thousands at a time. So the model ranks itself once per era, REPLACING rather than accumulating, which makes the weights describe the current weights by construction. ESTIMATOR. Not WinRateFromCounts(): it returns NO_DATA below 100 raw trades BEFORE shrinking, which here would fire on every tier every era and hand all four the pooled rate - the tiers could never separate and the mechanism would be inert. Shrinkage is the answer to a small sample; a floor in front of it means the shrinkage never runs. Instead: a Beta prior of TIER_PRIOR_EFF_N pseudo-observations centred on the model's pooled holdout rate, counted in EFFECTIVE observations, because overlapping triple-barrier labels mean 800 raw fires can be worth ~12 independent ones. Rounded to the integer, not to the decade NormalizeWinRate() uses, which would collapse the shrunk tiers back into one number. NO SAME-ERA CIRCULARITY, and it falls out of the ordering rather than a guard: weights are computed at the END of era N, so the vote scored during era N was cast with era N-1's weights. The deploy gate never grades a vote whose weights were fitted on the bars it is scoring. Residual leakage remains - the same OOS bars each era under a different model - and is stated in the code rather than papered over. Both DB clobber paths are closed: ApplyPatternWeight() declines once self-ranked, and UpdateSignalsWeights()' filter.Weight() call is guarded by SelfRanked() - guarding only the tiers would have let the hourly ranking pass undo half the self-ranking. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 16:00:32 -04:00
//--- Direct tier setter, deliberately NOT routed through ApplyPatternWeight(): that override
//--- declines writes once self-ranking is live, which is exactly what must not happen to the
//--- self-ranker's own writes. Two doors, because they serve opposite purposes.
void ApplyTierWeight(const int tier, const int weight)
{
switch(tier)
{
case 0: Pattern_0(weight); break;
case 1: Pattern_1(weight); break;
case 2: Pattern_2(weight); break;
default: Pattern_3(weight); break;
}
}
fix(build): four compile faults - one was a SILENT enum collision that inverted the direction policy Reported by the user's MetaEditor compile of f64e0f8 (26 errors, 4 warnings). The four warnings mattered more than the errors. 1. INTELLIGENT WAS TWO ENUMS. MONEY_MANAGEMENT_STRATEGY::INTELLIGENT (=1) is declared BEFORE TRADING_DIRECTION::INTELLIGENT (=3) in InputEnums.mqh, so MQL5 resolved every 'tradingdirection == INTELLIGENT' to the MM member and converted it to value 1 = TRADING_DIRECTION::LONG_ONLY. Wrong in both directions at once: selecting Intelligent (3) matched NOTHING and silently traded both sides, while selecting Long only (1) matched and handed the decision to the measured drift verdict - which can answer SHORT_ONLY, so the one setting that must never go short could have. Reported by the compiler as a WARNING only, never an error. Renamed to DIRECTION_INTELLIGENT; the VALUE stays 3, so saved .set files are unaffected. Swept every enum in the repo for sibling collisions (38 enums, detector validated against the pre-fix source, which it flags): none remain. 2. g_warriorMetaGate sits above the class it points at - added the forward declaration, the same pattern g_warriorEnsemble already uses in ExpertSignalAIBase.mqh. 3. The broker-time rename (b63e39f) never reached BufferNewTickSignal's PARAMETER or its two call sites: the local became brokerTime, the parameter stayed gmtTime, and the body was rewritten to read brokerTime. All five sites now agree. 4. ConfigureAISignal calls IsMetaTarget() from a free function - moved it to the public section (identity, not an implementation seam); the other meta seams stay protected. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 13:16:45 -04:00
//--- Which target this model trains toward: true = the meta head (trade-quality over fired
//--- candidates), false = a per-bar direction model. Reads the constructor-set target; nothing
//--- can flip it.
fix(build): four compile faults - one was a SILENT enum collision that inverted the direction policy Reported by the user's MetaEditor compile of f64e0f8 (26 errors, 4 warnings). The four warnings mattered more than the errors. 1. INTELLIGENT WAS TWO ENUMS. MONEY_MANAGEMENT_STRATEGY::INTELLIGENT (=1) is declared BEFORE TRADING_DIRECTION::INTELLIGENT (=3) in InputEnums.mqh, so MQL5 resolved every 'tradingdirection == INTELLIGENT' to the MM member and converted it to value 1 = TRADING_DIRECTION::LONG_ONLY. Wrong in both directions at once: selecting Intelligent (3) matched NOTHING and silently traded both sides, while selecting Long only (1) matched and handed the decision to the measured drift verdict - which can answer SHORT_ONLY, so the one setting that must never go short could have. Reported by the compiler as a WARNING only, never an error. Renamed to DIRECTION_INTELLIGENT; the VALUE stays 3, so saved .set files are unaffected. Swept every enum in the repo for sibling collisions (38 enums, detector validated against the pre-fix source, which it flags): none remain. 2. g_warriorMetaGate sits above the class it points at - added the forward declaration, the same pattern g_warriorEnsemble already uses in ExpertSignalAIBase.mqh. 3. The broker-time rename (b63e39f) never reached BufferNewTickSignal's PARAMETER or its two call sites: the local became brokerTime, the parameter stayed gmtTime, and the body was rewritten to read brokerTime. All five sites now agree. 4. ConfigureAISignal calls IsMetaTarget() from a free function - moved it to the public section (identity, not an implementation seam); the other meta seams stay protected. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 13:16:45 -04:00
bool IsMetaTarget(void) const { return m_trainTarget == 1; }
feat(rank): AI models rank their own confidence tiers from held-out outcomes Closes the caveat 4858507 shipped with: the vote is a confidence percentage, but only to the extent the pattern weights are measured. AI tier weights sat at their designed defaults (25/50/75/100) because AI rows only ever arrive from LIVE journaling, of which a training run produces almost none. AND A STALE-TIER BUG THAT MADE THE EVIDENCE MEANINGLESS. The OOS scan bucketed every scanned bar by ConfidenceTier(), which reads dPrevSignal - and dPrevSignal is assigned in PASS 1 only, never anywhere in the OOS scan. So an entire era's fires were bucketed by one stale, unrelated bar's confidence and landed in a SINGLE tier. That is the "tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0)" symptom recorded on 2026-08-16 and attributed to the calibration clamp. The clamp was real and was fixed then; this is a second, independent cause of the identical output that survived that fix untouched - which is why the log kept reading the same afterwards. Two causes, one symptom. Now ConfidenceTierFor(adjSig): the bar this iteration actually scored. WHY THIS DOES NOT WRITE ROWS TO THE SIGNAL DB, which was the obvious reading of "fill the database during training". The user's own observation is the reason: a classic Pattern_2 is a fixed geometric condition, so its win rate is legitimately accumulated over years, but an AI Pattern_2 means "confidence landed in tier 2" and tier 2 under era 100's weights is a different statement from tier 2 under era 500's. The DB's value is ACCUMULATION, and accumulation is exactly what is wrong here - it would average together models that no longer exist, while colliding with the per-table row cap and mixing measured-on-holdout outcomes into the live ledger's own tables. What the DB actually supplies is a measured win rate per pattern, and pass 3 already computes that on held-out bars, thousands at a time. So the model ranks itself once per era, REPLACING rather than accumulating, which makes the weights describe the current weights by construction. ESTIMATOR. Not WinRateFromCounts(): it returns NO_DATA below 100 raw trades BEFORE shrinking, which here would fire on every tier every era and hand all four the pooled rate - the tiers could never separate and the mechanism would be inert. Shrinkage is the answer to a small sample; a floor in front of it means the shrinkage never runs. Instead: a Beta prior of TIER_PRIOR_EFF_N pseudo-observations centred on the model's pooled holdout rate, counted in EFFECTIVE observations, because overlapping triple-barrier labels mean 800 raw fires can be worth ~12 independent ones. Rounded to the integer, not to the decade NormalizeWinRate() uses, which would collapse the shrunk tiers back into one number. NO SAME-ERA CIRCULARITY, and it falls out of the ordering rather than a guard: weights are computed at the END of era N, so the vote scored during era N was cast with era N-1's weights. The deploy gate never grades a vote whose weights were fitted on the bars it is scoring. Residual leakage remains - the same OOS bars each era under a different model - and is stated in the code rather than papered over. Both DB clobber paths are closed: ApplyPatternWeight() declines once self-ranked, and UpdateSignalsWeights()' filter.Weight() call is guarded by SelfRanked() - guarding only the tiers would have let the hourly ranking pass undo half the self-ranking. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 16:00:32 -04:00
//--- True once this model has measured its own tier win rates on held-out bars. While true the
//--- signal DB's ranking is declined for this filter - see ApplyPatternWeight's comment and
//--- CExpertSignalCustom::SelfRanked().
virtual bool SelfRanked(void) const override { return m_tiersSelfRanked; }
feat(vote): CONSENSUS arithmetic - agreement is now what the threshold dials Era-680 report, all three observations one equation: "peak 29, no arrows at threshold 30" / "at 20, arrows on EVERY bar" / "label at 12 while arrows everywhere". Under the voters-only divisor, any bar with at least one directional voter read the weighted mean of the firing tiers' weights - and once the tiers self-ranked to each model's pooled win rate (~28-31), that mean was NEAR-CONSTANT regardless of headcount. One member alone: ~29. Four unanimous: ~29. Min_Vote_Open was a step function around that constant - above it nothing ever fired, below it everything did - and the label's 12 was a 3v1 split netting through the same divisor. Not three display bugs: one arithmetic that could not express agreement. The divisor is now the CAPABLE weight - every filter that could vote, whether it did or not: * live (Direction): VoteCapableWeight() - classic pattern ladders always, veto filters never, AI members once past the same readiness test LongCondition gates on. A model still training must not dilute an ensemble it cannot join: four trainees + one deployed model is a solo chart wearing an ensemble label, and the solo vote reads full strength. * gate (EnsembleEraVerdict): g_ensVoteWeightSum accumulates for every member that EVALUATED the bar, Neutral included. * overlay sweep + prospective readout: weight counts whenever the member has data; a snapshotted Neutral dilutes. One arithmetic, four sites, same numbers everywhere. What the numbers become (four members, w~0.29, tiers~29): unanimous ~29 - the CEILING, which is the pooled win rate and is what the peak displays; 3-of-4 ~22; 2-of-4 ~14.5; 3v1 ~14.5. Min_Vote_Open 20 now means "roughly three-quarters of the ensemble's trust agrees, net". It MUST sit below the ceiling to ever fire - the census/peak states the ceiling. This is the ensemble the user specified in the original design discussion ("if the perceptron also votes, both together reach the threshold; if another NN votes the other side, the threshold is not reached") - union semantics was the pre-ensemble behaviour, kept until measurement showed its vote magnitude was a constant. Plus overlay DECLUSTERING, the other half of "arrows on every bar": the same three NMS rules as the per-member arrows (same-direction runs collapse to their first bar, cross-direction flicker keeps the stronger side), online over the sweep's strictly oldest->newest walk. Suppression is a verdict and deletes a standing arrow; the den==0 no-data skip still never does. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 06:58:03 -04:00
//--- An AI member's say in the consensus denominator: its module weight once it is ALLOWED to
//--- vote (the same readiness test LongCondition gates on), zero before that.
feat(vote): CONSENSUS arithmetic - agreement is now what the threshold dials Era-680 report, all three observations one equation: "peak 29, no arrows at threshold 30" / "at 20, arrows on EVERY bar" / "label at 12 while arrows everywhere". Under the voters-only divisor, any bar with at least one directional voter read the weighted mean of the firing tiers' weights - and once the tiers self-ranked to each model's pooled win rate (~28-31), that mean was NEAR-CONSTANT regardless of headcount. One member alone: ~29. Four unanimous: ~29. Min_Vote_Open was a step function around that constant - above it nothing ever fired, below it everything did - and the label's 12 was a 3v1 split netting through the same divisor. Not three display bugs: one arithmetic that could not express agreement. The divisor is now the CAPABLE weight - every filter that could vote, whether it did or not: * live (Direction): VoteCapableWeight() - classic pattern ladders always, veto filters never, AI members once past the same readiness test LongCondition gates on. A model still training must not dilute an ensemble it cannot join: four trainees + one deployed model is a solo chart wearing an ensemble label, and the solo vote reads full strength. * gate (EnsembleEraVerdict): g_ensVoteWeightSum accumulates for every member that EVALUATED the bar, Neutral included. * overlay sweep + prospective readout: weight counts whenever the member has data; a snapshotted Neutral dilutes. One arithmetic, four sites, same numbers everywhere. What the numbers become (four members, w~0.29, tiers~29): unanimous ~29 - the CEILING, which is the pooled win rate and is what the peak displays; 3-of-4 ~22; 2-of-4 ~14.5; 3v1 ~14.5. Min_Vote_Open 20 now means "roughly three-quarters of the ensemble's trust agrees, net". It MUST sit below the ceiling to ever fire - the census/peak states the ceiling. This is the ensemble the user specified in the original design discussion ("if the perceptron also votes, both together reach the threshold; if another NN votes the other side, the threshold is not reached") - union semantics was the pre-ensemble behaviour, kept until measurement showed its vote magnitude was a constant. Plus overlay DECLUSTERING, the other half of "arrows on every bar": the same three NMS rules as the per-member arrows (same-direction runs collapse to their first bar, cross-direction flicker keeps the stronger side), online over the sweep's strictly oldest->newest walk. Suppression is a verdict and deletes a standing arrow; the den==0 no-data skip still never does. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 06:58:03 -04:00
virtual double VoteCapableWeight(void) override
{
feat(ensemble): per-NN inputs replace the preset selector - the meta head becomes the vote's gate User design (2026-08-19): 'remove the enum menu that selects neural networks... individual inputs for every NN just like classic signals... the META NN should be integrated into the voting decision pipeline when enabled... as a bonus meta labelling is applied to enabled NNs.' - AI_CHOICE is GONE (tombstoned per the stale-.set doctrine). Use_MLP/Use_CONV/Use_LSTM/ Use_CONVLSTM are ordinary bools like the classic votes; the ensemble arithmetic adapts to any subset because the consensus divisor is the enabled capable weight. Two or more enabled = ensemble (|ENS1 token + joint gate, exactly the old AI_HYBRID fingerprints, so existing weight files keep loading); one = the old solo preset; none = classic-only. - Use_MetaLabeling un-couples META from the direction NNs (the old selector made them mutually exclusive). S3 ships: CSignalMETA::LiveMetaGate scores each vote-cleared entry (shared window at bar 1 + proposal descriptor: side, net vote, live geometry, spread/ATR; pattern one-hot ZEROED - ranking, not calibrated probability, documented in the body) and vetoes below the cost-adjusted break-even. Entries only; fail-open everywhere, loudly. - COEXISTENCE HAZARDS closed: VoteCapableWeight()=0 and ProspectiveVote()=false for the meta target - solo-only until today, a trained META would otherwise sit in the consensus divisor as a permanent abstainer and shrink every vote by its module weight. - CERTIFIED == TRADED: the ensemble era verdict replays the identical veto through the same g_warriorMetaGate pointer over its OOS fired bars (bar re-resolved from the row's own time; fail-open counted as fires and reported: 'metaGate: N approved, M vetoed, K unscored'). The overlay deliberately does NOT replay it (veto-filter-in-replay class, calendar-cliff precedent) - documented at the sweep site. Solo charts' own gate does not model the veto - the standing solo-gate caveat, documented at the input. - DB continuity: the pattern/journal DB fingerprint's first slot was (int)AIType; DbLegacyAiSlot() maps every legacy-expressible config to its OLD value (new 2-3 member subsets get 100+bitmask, outside the legacy range) so no existing database re-keys. filterID becomes the enabled roster via one EnabledNNSummary(). - HUD: the meta line shows the gate (armed/(trn), last P vs BE, ok/veto tally); the armed/disarmed announcement fires on state change via one latch (MetaGateArmedNow), not only when an entry happens to be proposed. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 13:01:02 -04:00
//--- The meta head is a GATE, not a voter (its Long/ShortCondition are structurally 0).
//--- Now that it can coexist with direction members (2026-08-19), counting it here would
//--- park a permanent abstainer in the consensus divisor and shrink every vote by its
//--- module weight - a member that can never agree must not dilute the agreement measure.
if(IsMetaTarget())
return 0.0;
feat(vote): CONSENSUS arithmetic - agreement is now what the threshold dials Era-680 report, all three observations one equation: "peak 29, no arrows at threshold 30" / "at 20, arrows on EVERY bar" / "label at 12 while arrows everywhere". Under the voters-only divisor, any bar with at least one directional voter read the weighted mean of the firing tiers' weights - and once the tiers self-ranked to each model's pooled win rate (~28-31), that mean was NEAR-CONSTANT regardless of headcount. One member alone: ~29. Four unanimous: ~29. Min_Vote_Open was a step function around that constant - above it nothing ever fired, below it everything did - and the label's 12 was a 3v1 split netting through the same divisor. Not three display bugs: one arithmetic that could not express agreement. The divisor is now the CAPABLE weight - every filter that could vote, whether it did or not: * live (Direction): VoteCapableWeight() - classic pattern ladders always, veto filters never, AI members once past the same readiness test LongCondition gates on. A model still training must not dilute an ensemble it cannot join: four trainees + one deployed model is a solo chart wearing an ensemble label, and the solo vote reads full strength. * gate (EnsembleEraVerdict): g_ensVoteWeightSum accumulates for every member that EVALUATED the bar, Neutral included. * overlay sweep + prospective readout: weight counts whenever the member has data; a snapshotted Neutral dilutes. One arithmetic, four sites, same numbers everywhere. What the numbers become (four members, w~0.29, tiers~29): unanimous ~29 - the CEILING, which is the pooled win rate and is what the peak displays; 3-of-4 ~22; 2-of-4 ~14.5; 3v1 ~14.5. Min_Vote_Open 20 now means "roughly three-quarters of the ensemble's trust agrees, net". It MUST sit below the ceiling to ever fire - the census/peak states the ceiling. This is the ensemble the user specified in the original design discussion ("if the perceptron also votes, both together reach the threshold; if another NN votes the other side, the threshold is not reached") - union semantics was the pre-ensemble behaviour, kept until measurement showed its vote magnitude was a constant. Plus overlay DECLUSTERING, the other half of "arrows on every bar": the same three NMS rules as the per-member arrows (same-direction runs collapse to their first bar, cross-direction flicker keeps the stronger side), online over the sweep's strictly oldest->newest walk. Suppression is a verdict and deletes a standing arrow; the den==0 no-data skip still never does. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 06:58:03 -04:00
if(!m_trainingComplete && !(m_inferenceOnly && m_modelLoadedFromDisk))
return 0.0;
return ModuleWeight();
}
//--- Gate for the settled PER-ERA diagnostics - see TRAIN_LOG_EVERY_ERAS. HOW OFTEN A DIRECTION
//--- ACTUALLY OCCURS, as a percentage of labelled bars. Prints that mark a state CHANGE (new
//--- best, stage transition, restore, deploy verdict, warning) must never be put behind this; it
//--- exists only for the lines that repeat with the era heartbeat.
feat(calibration): fit the operating point on the label rate instead of on edge The margin threshold now sits where the model calls a direction as often as a direction actually occurs. Nothing else. WHY THE OLD OBJECTIVE HAD TO GO. It maximised `coverage x (precision - breakEven)`, and this function's own comments were already the case against it: over 98 consecutive fits of the shipped SP500 H4 model, correlation between the chosen threshold and the win rate at it was -0.056, while the era-to-era spread of that win rate (1.32pp) matched its own binomial SE (1.25pp) to within 0.07pp. The margin does not rank trades. So the argmax returned whichever of ~37 bins drew the luckiest sample, and the threshold teleported 0.42 -> 0.04 -> 0.74 in three eras. The response at the time was to build a null-of-the-maximum gate, an effective-sample SE and a parsimony fallback to hold the noise down. All of that is gone now, because fitting on calibration removes the problem instead of bounding it: coverage is a ratio against a fixed denominator so it is well determined at every bin, the target is a measured label rate rather than an outcome, and nothing is maximised over a noisy curve so there is no best-of-N to correct for. Net 174 lines out, 62 in. It deliberately does not chase edge. It cannot - at ~0 measured edge no operating point has more of it, and pretending otherwise is what produced a threshold of 0.96 that still passed 60% of bars while the model called a direction ~10x too often. The edge at the chosen point is still REPORTED, just no longer what chooses it. THREE READINGS OF ONE QUANTITY, AND THEY DISAGREE. "How often does a direction occur" is measured in three places and gives ~7% (the scan's own tally), ~41% (the era loop's counters, via this function's old coverage floor) and ~50% (the ensemble gate's OOS base rate). They cannot all be right. Rather than pick one silently, ScanDirectionalRatePct() and EraDirectionalRatePct() are now named accessors, the fitter targets the SCAN - that is the tally the operator reads, and the one "predict the labels as measured during the scan phase" names - and the threshold line PRINTS BOTH every time it moves, so the disagreement is on the record instead of buried in a derived floor. The ensemble gate's own floor is deliberately NOT changed in this commit. If the scan is right, a calibrated member covering ~7% of bars cannot clear a 12.4% floor and every model would fail the gate by construction; if the gate is right, the scan tally is wrong. The CALIBRATION field added in 667f2bc reports the OOS true class rates directly and settles it in one era - that measurement comes first, and the floor follows it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 10:20:31 -04:00
double ScanDirectionalRatePct(void) const
{
long tot = (long)m_labelPrebuildBuyCount + m_labelPrebuildSellCount + m_labelPrebuildNeutralCount;
if(tot <= 0)
return -1.0;
return 100.0 * (double)(m_labelPrebuildBuyCount + m_labelPrebuildSellCount) / tot;
}
double EraDirectionalRatePct(void) const
{
long tot = (long)m_trueBuyCount + m_trueSellCount + m_trueNeutralCount;
if(tot <= 0)
return -1.0;
return 100.0 * (double)(m_trueBuyCount + m_trueSellCount) / tot;
}
//--- BREAK-EVEN THAT KNOWS ABOUT THE HORIZON. CostAdjustedBreakEvenPct is risk/(risk+reward):
//--- the win rate a trade needs when it is CERTAIN to end at one barrier or the other.
feat(breakeven): the break-even every layer scores against prices a trade that always resolves CostAdjustedBreakEvenPct is risk/(risk+reward) and has no horizon term. It is the win rate a trade needs when it is CERTAIN to end at one barrier or the other. SimulateTradeOutcome has an explicit branch for the case where it does not - runs out of horizon, closes at the last bar seen for whatever P&L that is - so on this label geometry the figure describes a different trade than the one being replayed. The gap is measurable and large. Across 21 exit replays today on SP500 H4 the geometric figure read 34.5% while the EA's own R simulation crossed zero between 27.4% (lowest positive) and 28.9% (highest non-positive). Independent corroboration: the zero-skill reference, computed empirically over every scored bar as max(winLong,winShort)/bars, reads 25.4% - add cost and it lands on the same ~28%. The geometric number is the outlier, and every edge printed against it was ~6.5pp too pessimistic: LSTM's 30.6%-win era reported -4.0pp while its replay returned +0.075 R on the same trades. With a timeout share t paying a mean m R apiece, expectancy is w(1+RR) + t(1+m) - 1, so w* = (1 - t(1+m)) / (1 + RR) = CostAdjustedBreakEvenPct x (1 - t(1+m)) which needs no new geometry - the existing figure already carries 1/(1+RR). This commit MEASURES ONLY. The replay now separates timeout exits from barrier exits and latches t and m for the next era to read (the accumulators are zeroed at era start and filled at era end, so a mid-era reader sees zero trades and would fall back forever). Both break-evens print side by side on the replay line with t and m beside them, and the threshold line's REPORTED edge - which selects nothing - switches to the horizon-aware figure so the operator stops reading a wrong sign. DELIBERATELY NOT CHANGED: LiveMetaGate's veto and the rung selector's BarrierMinReachPct still read the geometric value. Both are decisions - the second re-derives geometry and therefore relabels - and t and m have so far only been inferred from a zero-crossing, never seen on a log. One era of this instrumentation settles that. The file already contained the argument, one branch away, in the vote-exit comment: a vote exit produces a CONTINUOUS payoff, not a win or a loss, and that is why an exit-aware gate cannot go on scoring win-rate against a fixed break-even. A horizon timeout is the same thing, and unlike vote exits it is on by default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 19:26:25 -04:00
double EmpiricalBreakEvenPct(void)
{
double geom = CostAdjustedBreakEvenPct();
double t = m_lastTimeoutShare;
double m = m_lastTimeoutMeanR;
//--- Prefer the era in flight once its own replay has completed; otherwise the latched one.
if(m_simTrades > 0)
{
t = (double)m_simTimeouts / m_simTrades;
m = (m_simTimeouts > 0) ? m_simTimeoutRSum / m_simTimeouts : 0.0;
}
if(t < 0.0)
return geom;
double adj = geom * (1.0 - t * (1.0 + m));
//--- A timeout mean below -1 R is not reachable (the stop would have taken it first), so this
//--- cannot go negative from honest inputs. Clamped anyway: a break-even at or below zero would
//--- read as "any win rate pays", which is never a true statement about a trade.
return (adj > 0.0 && adj <= geom) ? adj : geom;
}
revert(labels): drop the one-sided exit target; measure the calibration drift instead Reverts a863796 on the operator's call - "unnecessary complexity". It was right about the mechanism and wrong about the priority: it re-cut the classes for a case the measured verdict never reaches (SP500 H4 reads "both sides" at the derived geometry), while the drift that IS happening affects every chart and every era. Recoverable from a863796 if a one-sided book ever becomes real. Two pieces of it survive, both independent of the exit idea: The drift verdict keeps reading m_winLongCache/m_winShortCache rather than the collapsed label pair. That line reports always-long vs always-short win rates, which is what the win caches hold - each side scored on its own barriers, published before the collapse. The label pair carries only the side touched first, so it undercounted long wins by the both-won-goes-to-short share. There are zero both-won bars at any geometry with target >= stop, so this changes no number today; it changes the wrong number to the right one. And the .cfg gains nothing and loses nothing: the two appended ints go away again, and they were the last fields, so a .cfg written by yesterday's build still reads correctly - the loader simply stops before them. WHAT THE REVERT MAKES ROOM FOR. The operator's actual requirement is that the model reproduce the label distribution the scan measured, and nothing in the pipeline ties it to that. The loss trains on a rebalanced sample and the abstain rate is owned by a margin threshold fitted on EDGE, so the call rate and the label prior can drift arbitrarily far apart - and did, invisibly: at era 1350 the models call Buy on 20-28% and Sell on 22-32% of bars against a scan-measured 2.1% and 4.8%. Roughly a 10x over-call, and not one line in the journal said so. The era line now carries it: CALIBRATION calls vs true rate Buy 28% vs 2% (14.0x) Sell 32% vs 5% (6.4x) Neutral 40% vs 93% (0.4x) Reported as a ratio because that is the readable number - 1.0x is calibrated. This is deliberately a measurement and not yet a correction: matching the label rate would put coverage near 7%, below the ensemble gate's own 12.4% coverage floor, so calibration and the gate are in direct conflict and which one yields is the operator's call, not mine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 10:14:26 -04:00
//--- "3.2x" / "0.4x" / "n/a" for a predicted-vs-true class rate pair. Both are already rounded
//--- percentages, so a true rate of 0 has no ratio to report rather than an infinite one.
string CalibrationRatio(const int predPct, const int truePct) const
{
if(predPct < 0 || truePct <= 0)
return "n/a";
return StringFormat("%.1fx", (double)predPct / (double)truePct);
}
feat(logs): throttle the settled per-era diagnostics - measured 22MB/9.5h of confirmed-working systems Measured from the journal (2026-08-19): the era deep-dive line (~2KB) plus the excursion verdict, tier re-rank, calibration move, barrier hold and selection-regressed note each printed EVERY era for EVERY member - ~940 eras/member/day - long after the systems they watch were confirmed working. Yesterday's file was 1.3GB (70% of it the news-filter calendar spam the sweep fix already removed). VerboseMode returns as an INPUT (demoted 2026-08-01 for the marketplace; that track is dead since the 2026-08-16 pivot) and gains a second job: false throttles each settled per-era print to eras 0-3 plus every TRAIN_LOG_EVERY_ERAS-th (25 ~= one deep-dive per ~15min per member); true restores the per-era firehose, flippable live. Never throttled: anything that marks a CHANGE - new bests, restores + eta decays, plateau stage transitions, deploy approvals, warnings, errors, the label-cache/adoption one-shots, and the combined-vote gate line (the active system's primary telemetry, still every era). Semantic fixes over blanket gating: - barrier hold now ARMS silently and prints only when the hold outlasts the 2-min report interval - a brief hold every era is the design, the long hold is the watchdog case the line exists for; - the ensemble deploy REFUSAL prints immediately when its reason changes (that is a finding), on cadence when unchanged; - the filtered-view census prints when its RESULT moves (drawn count, or strongest vote by >=2pp) and at least every 10th sweep. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 09:36:15 -04:00
bool TrainLogDue(void) const
{
return VerboseMode || m_eraCount <= 3 || (m_eraCount % TRAIN_LOG_EVERY_ERAS == 0);
}
feat(chart): reconstruct the filtered view behind the handover point Completes the filtered view from 282b535, which only reached forward of attach. On a multi-hour training run that is the entire time you are looking at the chart, so the answer to "how would the whole bot have traded" was blank exactly when it was wanted. The sweep lives on the AGGREGATE signal, which is the only object holding every filter. AI members contribute their CACHED per-bar decision from the era scan - no inference re-runs, the cache already spans the chart - and the classic ladders are replayed with EvalShift(i), the same mechanism CSignalMETA's candidate sweep uses and exact because every classic pattern condition anchors on StartIndex(). Combination is the live one: weighted mean over voting filters, abstentions out of both sums, against Min_Vote_Open. THE REPLAY CORRUPTS LIVE JOURNALING IF LEFT UNGUARDED, and this is the part that is not obvious. Live journaling reads m_active_pattern_long/short from the PREVIOUS Direction() call. Replaying hundreds of past bars between two live bars leaves those slots holding whichever bar the sweep stopped on, so the next live bar journals that pattern under the current timestamp - a corrupted row in the very table pattern win rates are computed from, which is now also where vote weights come from. Save/RestoreVoteState() brackets every replayed call. CSignalMETA gets away without it only because its sweep runs once, at the first era, before any of that state matters. TWO SOURCES OF TRUTH, KEPT APART. A reconstruction cannot know the broker rejected an order - it has no stops level, ATR warm-up or swing-history sync as they were at that moment - so it is an upper bound: honest about the vote, optimistic about placement. It therefore stops dead at the handover bar, which is latched ONCE so later rebuilds cannot creep it forward and start overwriting real decisions with guesses, and its arrows say "reconstructed (vote only - order validation not replayed)" in the tooltip. Someone comparing two arrows either side of that line has to be able to tell which is a record and which is a replay, and the chart is the only place they look. Re-armed on any era boundary (summed era counters), because that is when the answer changes - RankTiersFromOos has just re-derived every tier's vote weight - and only between sweeps, so a restart cannot leave the previous pass's tail undrawn. Chunked at 150 bars per timer slice: each bar replays Direction() on every classic filter, which is real indicator work on the chart thread, and an unchunked sweep here is the 2026-07-26 arrow-restore freeze waiting to happen. SIGNAL_RESCAN_LOOKBACK_BARS moves to ExpertSignalCustom.mqh alongside SIG_ARROW_PREFIX - same include-order reason, and the two rebuilds should reach the same distance or the raw and filtered views are not comparable. Known gap: on a classic-only chart the reconstruction is built once and not refreshed when the hourly DB ranking moves the classic weights. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 16:05:13 -04:00
//--- This model's cached decision for bar `idx`, already converted to the signed vote it would
//--- have cast. Body in AIBase\ChartUI.mqh.
feat(hud): per-member neuron lines + a vote label that moves as the nets learn Both 2026-08-19 reports were the same staleness: every source behind the label was an ERA artifact (live cache refills at pass-3 completion, the snapshot copies once per era, dPrevSignal is the frozen purge-band edge bar) - so the readout stepped at era cadence at best, stayed glued to one direction, and lagged the era counter. DisplayInference(): throttled (4s, 1s across an era boundary), SIDE-EFFECT-FREE forward of the current decision bar (window ending on bar 1, same question the live path asks) through the LEARNER net. Batch-norm running stats are bracketed frozen/RESTORED via the new CNet::GetBatchNormFrozen() + CNeuronBatchNormOCL::StatsFrozen() - restore, not unfreeze, because a display tick can land between pass-3 chunks whose whole scan holds them frozen. Writes nothing a trading or training path reads (dPrevSignal, NMS state, tallies, watermarks all untouched; RefreshLatestSignal is not reusable here precisely because it writes all of them). LSTM safe by construction: h/c zeroed per forward. ProspectiveVote() reads the fresh forward as its FIRST source; the era-artifact chain becomes the fallback (meta head, warm-up, window holes). DisplayHudLine(): the reference library's training label, per ensemble member - name, output activations (softmax probs or raw scalar), the decision, its weighted vote (the exact consensus numerator term), era, recent average error, "(trn)" while not vote-capable. Rendered under the vote line in RefreshVoteReadout BEFORE the live-vote defer (member lines are telemetry, not tradable readings), coloured by the member's own direction in muted tones - the vote line's strict green-only-when-it-would-trade rule is untouched. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 08:48:34 -04:00
virtual string DisplayHudLine(void) override;
//--- What this model would vote on the CURRENT bar if it were deployed. The readiness gate in
//--- LongCondition() is what this bypasses, and ONLY for display: dPrevSignal is the decision,
//--- deployed or not.
feat(chart): show the PROSPECTIVE vote while the models are still training The readout sat at "VOTE 0.0%, 0 voters" constantly. Correct, and useless. LongCondition()/ShortCondition() return 0 behind the readiness gate for the entire training run - a model that is not deployed does not vote - so the LIVE vote is structurally zero for hours, which is exactly the period the readout is being watched. Worse, it was the same display whether the models were silent, undeployed, or the filter list was empty: three different situations, one number. When no filter casts a real vote, the readout now shows the PROSPECTIVE one - what these models are saying right now, through the identical tier/weight arithmetic, minus the readiness gate. That is the same quantity the historical overlay reconstructs on cached bars, deliberately, so the live line and the reconstructed arrows are the same measure and can be read against each other. It can never be mistaken for a decision: labelled "-> training, not tradable yet", drawn dimmer than "no trade", and `fires` is forced false regardless of magnitude, because saying "-> TRADE" about a number that cannot place an order is the precise overstatement this readout exists to prevent. m_direction is untouched - display only, no trading path reads it. Confirms the sweep fix from 155f56e is live and working: "Filtered view: swept 4999 bar(s), 767 had a voter, drew 0 arrow(s). Strongest vote 36.0% against a 40.0% threshold." 4,999 bars against the previous 0. The remaining emptiness is the models, not the plumbing - see the reply for why lowering the threshold further is the wrong response to it. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 18:29:27 -04:00
virtual bool ProspectiveVote(double &signedVote, double &weight) override
{
signedVote = 0.0;
weight = 0.0;
feat(ensemble): per-NN inputs replace the preset selector - the meta head becomes the vote's gate User design (2026-08-19): 'remove the enum menu that selects neural networks... individual inputs for every NN just like classic signals... the META NN should be integrated into the voting decision pipeline when enabled... as a bonus meta labelling is applied to enabled NNs.' - AI_CHOICE is GONE (tombstoned per the stale-.set doctrine). Use_MLP/Use_CONV/Use_LSTM/ Use_CONVLSTM are ordinary bools like the classic votes; the ensemble arithmetic adapts to any subset because the consensus divisor is the enabled capable weight. Two or more enabled = ensemble (|ENS1 token + joint gate, exactly the old AI_HYBRID fingerprints, so existing weight files keep loading); one = the old solo preset; none = classic-only. - Use_MetaLabeling un-couples META from the direction NNs (the old selector made them mutually exclusive). S3 ships: CSignalMETA::LiveMetaGate scores each vote-cleared entry (shared window at bar 1 + proposal descriptor: side, net vote, live geometry, spread/ATR; pattern one-hot ZEROED - ranking, not calibrated probability, documented in the body) and vetoes below the cost-adjusted break-even. Entries only; fail-open everywhere, loudly. - COEXISTENCE HAZARDS closed: VoteCapableWeight()=0 and ProspectiveVote()=false for the meta target - solo-only until today, a trained META would otherwise sit in the consensus divisor as a permanent abstainer and shrink every vote by its module weight. - CERTIFIED == TRADED: the ensemble era verdict replays the identical veto through the same g_warriorMetaGate pointer over its OOS fired bars (bar re-resolved from the row's own time; fail-open counted as fires and reported: 'metaGate: N approved, M vetoed, K unscored'). The overlay deliberately does NOT replay it (veto-filter-in-replay class, calendar-cliff precedent) - documented at the sweep site. Solo charts' own gate does not model the veto - the standing solo-gate caveat, documented at the input. - DB continuity: the pattern/journal DB fingerprint's first slot was (int)AIType; DbLegacyAiSlot() maps every legacy-expressible config to its OLD value (new 2-3 member subsets get 100+bitmask, outside the legacy range) so no existing database re-keys. filterID becomes the enabled roster via one EnabledNNSummary(). - HUD: the meta line shows the gate (armed/(trn), last P vs BE, ok/veto tally); the armed/disarmed announcement fires on state change via one latch (MetaGateArmedNow), not only when an entry happens to be proposed. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 13:01:02 -04:00
//--- The meta head has no directional prospect - reporting one (even weight-only) would
//--- count it as a neutral voter in the prospective readout and dilute its denominator, the
//--- display twin of the VoteCapableWeight exclusion above. Its HUD line shows the gate.
if(IsMetaTarget())
return false;
//--- FRESH FORWARD FIRST. DisplayInference() asks the CURRENT weights the live question on a
//--- ~4s throttle instead.
feat(hud): per-member neuron lines + a vote label that moves as the nets learn Both 2026-08-19 reports were the same staleness: every source behind the label was an ERA artifact (live cache refills at pass-3 completion, the snapshot copies once per era, dPrevSignal is the frozen purge-band edge bar) - so the readout stepped at era cadence at best, stayed glued to one direction, and lagged the era counter. DisplayInference(): throttled (4s, 1s across an era boundary), SIDE-EFFECT-FREE forward of the current decision bar (window ending on bar 1, same question the live path asks) through the LEARNER net. Batch-norm running stats are bracketed frozen/RESTORED via the new CNet::GetBatchNormFrozen() + CNeuronBatchNormOCL::StatsFrozen() - restore, not unfreeze, because a display tick can land between pass-3 chunks whose whole scan holds them frozen. Writes nothing a trading or training path reads (dPrevSignal, NMS state, tallies, watermarks all untouched; RefreshLatestSignal is not reusable here precisely because it writes all of them). LSTM safe by construction: h/c zeroed per forward. ProspectiveVote() reads the fresh forward as its FIRST source; the era-artifact chain becomes the fallback (meta head, warm-up, window holes). DisplayHudLine(): the reference library's training label, per ensemble member - name, output activations (softmax probs or raw scalar), the decision, its weighted vote (the exact consensus numerator term), era, recent average error, "(trn)" while not vote-capable. Rendered under the vote line in RefreshVoteReadout BEFORE the live-vote defer (member lines are telemetry, not tradable readings), coloured by the member's own direction in muted tones - the vote line's strict green-only-when-it-would-trade rule is untouched. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 08:48:34 -04:00
if(DisplayInference())
{
signedVote = LiveVoteContribution(m_dispSignal);
weight = ModuleWeight();
return true;
}
//--- ERA-ARTIFACT FALLBACK CHAIN, newest first - reached only when the fresh forward above
//--- cannot run (meta head, warm-up, indicator hole).
fix(chart): the prospective vote was four models' opinion of ONE frozen bar "Still glued to buy." Verified in the pass structure rather than guessed: dPrevSignal is written ONLY by pass 1 (Training.mqh 1439/1442 - the sole assignment sites), and pass 1 SKIPS the feedForward for any bar a later pass will forward anyway (laterPassForwards) - which is the whole OOS window and the calibration band. Pass 3 forwards the newest bars every era but never writes dPrevSignal. Net effect: after every era, dPrevSignal holds the model's opinion of the newest PURGE-BAND EDGE BAR pass 1 happened to forward - one fixed mid-history bar, re-evaluated era after era. The readout was therefore showing four models' verdict on the same frozen bar, and that bar reads Buy. Glued to Buy, with flashes of Sell only while pass 1 was actively walking (the one window where dPrevSignal moves). ProspectiveVote() now reads the newest ARROW-CACHE entry first (walking back from the decision bar, bounded at 16), falling back to dPrevSignal only when the cache holds nothing. Pass 3 writes the adjusted decision for the newest (OOS) bars each era, so the cache's first non-sentinel entry is the model's most recent verdict on near-current data - and it is the same value the filtered overlay draws from, so the label and the reconstruction stay one quantity. A cached Neutral stops the walk: that is a real decision (vote 0, abstain -> shows in the "flat" count), not a missing one. Early in an era the cache is wiped to sentinel and everything falls through to dPrevSignal exactly as before, until pass 3 refills the newest rows. Expect the label to change per era now (as each pass 3 re-scores the newest bars under that era's weights), with the vote/flat split moving as members genuinely flip between direction and Neutral on recent data. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 21:47:51 -04:00
for(int idx = 1; idx <= 16; idx++)
{
if(CachedVoteAt(idx, signedVote))
{
weight = ModuleWeight();
return true;
}
}
fix(chart): display now reads era-end SNAPSHOTS - the live cache is wiped mid-era Full-pipeline analysis after "threshold 30, attained often, nothing drawn, still glued to buy". The log falsified the premise before any code did: 21:40:43 swept 4999, 794 voters, drew 491. Strongest 43.0% vs 30.0% 21:42:07 swept 4999, 0 voters, drew 0 21:51:30 swept 4999, 0 voters, drew 0 21:56:30 swept 4999, 922 voters, drew 382. Strongest 44.0% vs 30.0% The arrows WERE drawn - 491 of them, then 382 - and then erased. ONE root cause, three symptoms: every display path read m_arrowSignalCache, which is wiped to sentinel at each era start and only complete again when pass 3 finishes. With eras at ~30s and a sweep at ~17s: * ARROW FLICKER: a sweep landing mid-era found no voters anywhere, and its else-branch deleted the arrow on every voteless bar - erasing the previous sweep's entire output. The chart cycled populated -> blank -> populated; the user kept catching the blank phase. * READOUT GLUE: the newest-cache walk found only sentinel for ~90% of every era and fell through to dPrevSignal - the frozen purge-band edge bar that reads Buy. 659638e fixed which bar was frozen, not the freezing. * VOTER FLAP: 1299 -> 257 -> 1113 across back-to-back sweeps - each saw a different fraction of half-rebuilt caches. THE FIX, structural rather than another patch: 1. Era-end snapshots. RankTiersFromOos() runs at pass-3 completion - the one moment the cache is complete - and now copies it (raw signals, newest LOOKBACK+16 bars) into member-owned snapshot state, unconditionally, BEFORE its early return: an all-Neutral era is a snapshot worth showing, not an absence of one. Raw signals rather than votes, so a tier re-rank between eras reprices them at read time via LiveVoteContribution for free. 2. The sweep (SnapshotVoteAt) and the prospective readout both read snapshots; the readout's fallback chain is live-cache -> snapshot -> dPrevSignal, and the snapshot leg is the one that fires most of the time. 3. NO DATA IS NOT A VERDICT: a den==0 bar no longer deletes - only an actual sub-threshold vote takes an arrow down. This alone ends the wipe half of the flicker even where snapshots are missing (before the first era). 4. Arming moved from an era-counter diff (which fires at era BOUNDARIES, i.e. precisely when caches are about to be wiped) to g_warriorOverlayArmRequest, set by each RankTiersFromOos - "a member's snapshot just got fresher", the only event a redraw can act on. 60s rate limit collapses the four members' burst into one sweep. Classic-only charts arm once at start. 5. Census now reports the direction split - "922 had a voter (610 buy / 312 sell)" - so "the vote leans buy" is checkable from the log instead of inferred from arrow colours. Also visible in the log and worth knowing: the threshold flip-flopped 30 -> 40 -> 30 across the evening's re-inits (census lines at 21:42-21:51 ran at 40), so part of the observed blankness was configuration, not code. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 22:03:20 -04:00
//--- Era-end snapshot next - the fallback that actually fires for ~90% of every era, because
//--- the live cache above is wiped at era start and only refills when pass 3 completes.
fix(chart): display now reads era-end SNAPSHOTS - the live cache is wiped mid-era Full-pipeline analysis after "threshold 30, attained often, nothing drawn, still glued to buy". The log falsified the premise before any code did: 21:40:43 swept 4999, 794 voters, drew 491. Strongest 43.0% vs 30.0% 21:42:07 swept 4999, 0 voters, drew 0 21:51:30 swept 4999, 0 voters, drew 0 21:56:30 swept 4999, 922 voters, drew 382. Strongest 44.0% vs 30.0% The arrows WERE drawn - 491 of them, then 382 - and then erased. ONE root cause, three symptoms: every display path read m_arrowSignalCache, which is wiped to sentinel at each era start and only complete again when pass 3 finishes. With eras at ~30s and a sweep at ~17s: * ARROW FLICKER: a sweep landing mid-era found no voters anywhere, and its else-branch deleted the arrow on every voteless bar - erasing the previous sweep's entire output. The chart cycled populated -> blank -> populated; the user kept catching the blank phase. * READOUT GLUE: the newest-cache walk found only sentinel for ~90% of every era and fell through to dPrevSignal - the frozen purge-band edge bar that reads Buy. 659638e fixed which bar was frozen, not the freezing. * VOTER FLAP: 1299 -> 257 -> 1113 across back-to-back sweeps - each saw a different fraction of half-rebuilt caches. THE FIX, structural rather than another patch: 1. Era-end snapshots. RankTiersFromOos() runs at pass-3 completion - the one moment the cache is complete - and now copies it (raw signals, newest LOOKBACK+16 bars) into member-owned snapshot state, unconditionally, BEFORE its early return: an all-Neutral era is a snapshot worth showing, not an absence of one. Raw signals rather than votes, so a tier re-rank between eras reprices them at read time via LiveVoteContribution for free. 2. The sweep (SnapshotVoteAt) and the prospective readout both read snapshots; the readout's fallback chain is live-cache -> snapshot -> dPrevSignal, and the snapshot leg is the one that fires most of the time. 3. NO DATA IS NOT A VERDICT: a den==0 bar no longer deletes - only an actual sub-threshold vote takes an arrow down. This alone ends the wipe half of the flicker even where snapshots are missing (before the first era). 4. Arming moved from an era-counter diff (which fires at era BOUNDARIES, i.e. precisely when caches are about to be wiped) to g_warriorOverlayArmRequest, set by each RankTiersFromOos - "a member's snapshot just got fresher", the only event a redraw can act on. 60s rate limit collapses the four members' burst into one sweep. Classic-only charts arm once at start. 5. Census now reports the direction split - "922 had a voter (610 buy / 312 sell)" - so "the vote leans buy" is checkable from the log instead of inferred from arrow colours. Also visible in the log and worth knowing: the threshold flip-flopped 30 -> 40 -> 30 across the evening's re-inits (census lines at 21:42-21:51 ran at 40), so part of the observed blankness was configuration, not code. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 22:03:20 -04:00
if(m_prospectiveSigSnap != -2.0 && MathIsValidNumber(m_prospectiveSigSnap))
{
signedVote = LiveVoteContribution(m_prospectiveSigSnap);
weight = ModuleWeight();
return true;
}
fix(chart): the prospective vote was four models' opinion of ONE frozen bar "Still glued to buy." Verified in the pass structure rather than guessed: dPrevSignal is written ONLY by pass 1 (Training.mqh 1439/1442 - the sole assignment sites), and pass 1 SKIPS the feedForward for any bar a later pass will forward anyway (laterPassForwards) - which is the whole OOS window and the calibration band. Pass 3 forwards the newest bars every era but never writes dPrevSignal. Net effect: after every era, dPrevSignal holds the model's opinion of the newest PURGE-BAND EDGE BAR pass 1 happened to forward - one fixed mid-history bar, re-evaluated era after era. The readout was therefore showing four models' verdict on the same frozen bar, and that bar reads Buy. Glued to Buy, with flashes of Sell only while pass 1 was actively walking (the one window where dPrevSignal moves). ProspectiveVote() now reads the newest ARROW-CACHE entry first (walking back from the decision bar, bounded at 16), falling back to dPrevSignal only when the cache holds nothing. Pass 3 writes the adjusted decision for the newest (OOS) bars each era, so the cache's first non-sentinel entry is the model's most recent verdict on near-current data - and it is the same value the filtered overlay draws from, so the label and the reconstruction stay one quantity. A cached Neutral stops the walk: that is a real decision (vote 0, abstain -> shows in the "flat" count), not a missing one. Early in an era the cache is wiped to sentinel and everything falls through to dPrevSignal exactly as before, until pass 3 refills the newest rows. Expect the label to change per era now (as each pass 3 re-scores the newest bars under that era's weights), with the vote/flat split moving as members genuinely flip between direction and Neutral on recent data. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 21:47:51 -04:00
signedVote = 0.0;
feat(chart): show the PROSPECTIVE vote while the models are still training The readout sat at "VOTE 0.0%, 0 voters" constantly. Correct, and useless. LongCondition()/ShortCondition() return 0 behind the readiness gate for the entire training run - a model that is not deployed does not vote - so the LIVE vote is structurally zero for hours, which is exactly the period the readout is being watched. Worse, it was the same display whether the models were silent, undeployed, or the filter list was empty: three different situations, one number. When no filter casts a real vote, the readout now shows the PROSPECTIVE one - what these models are saying right now, through the identical tier/weight arithmetic, minus the readiness gate. That is the same quantity the historical overlay reconstructs on cached bars, deliberately, so the live line and the reconstructed arrows are the same measure and can be read against each other. It can never be mistaken for a decision: labelled "-> training, not tradable yet", drawn dimmer than "no trade", and `fires` is forced false regardless of magnitude, because saying "-> TRADE" about a number that cannot place an order is the precise overstatement this readout exists to prevent. m_direction is untouched - display only, no trading path reads it. Confirms the sweep fix from 155f56e is live and working: "Filtered view: swept 4999 bar(s), 767 had a voter, drew 0 arrow(s). Strongest vote 36.0% against a 40.0% threshold." 4,999 bars against the previous 0. The remaining emptiness is the models, not the plumbing - see the reply for why lowering the threshold further is the wrong response to it. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 18:29:27 -04:00
if(!MathIsValidNumber(dPrevSignal))
return false;
signedVote = LiveVoteContribution(dPrevSignal);
weight = ModuleWeight();
return true;
}
fix(chart): display now reads era-end SNAPSHOTS - the live cache is wiped mid-era Full-pipeline analysis after "threshold 30, attained often, nothing drawn, still glued to buy". The log falsified the premise before any code did: 21:40:43 swept 4999, 794 voters, drew 491. Strongest 43.0% vs 30.0% 21:42:07 swept 4999, 0 voters, drew 0 21:51:30 swept 4999, 0 voters, drew 0 21:56:30 swept 4999, 922 voters, drew 382. Strongest 44.0% vs 30.0% The arrows WERE drawn - 491 of them, then 382 - and then erased. ONE root cause, three symptoms: every display path read m_arrowSignalCache, which is wiped to sentinel at each era start and only complete again when pass 3 finishes. With eras at ~30s and a sweep at ~17s: * ARROW FLICKER: a sweep landing mid-era found no voters anywhere, and its else-branch deleted the arrow on every voteless bar - erasing the previous sweep's entire output. The chart cycled populated -> blank -> populated; the user kept catching the blank phase. * READOUT GLUE: the newest-cache walk found only sentinel for ~90% of every era and fell through to dPrevSignal - the frozen purge-band edge bar that reads Buy. 659638e fixed which bar was frozen, not the freezing. * VOTER FLAP: 1299 -> 257 -> 1113 across back-to-back sweeps - each saw a different fraction of half-rebuilt caches. THE FIX, structural rather than another patch: 1. Era-end snapshots. RankTiersFromOos() runs at pass-3 completion - the one moment the cache is complete - and now copies it (raw signals, newest LOOKBACK+16 bars) into member-owned snapshot state, unconditionally, BEFORE its early return: an all-Neutral era is a snapshot worth showing, not an absence of one. Raw signals rather than votes, so a tier re-rank between eras reprices them at read time via LiveVoteContribution for free. 2. The sweep (SnapshotVoteAt) and the prospective readout both read snapshots; the readout's fallback chain is live-cache -> snapshot -> dPrevSignal, and the snapshot leg is the one that fires most of the time. 3. NO DATA IS NOT A VERDICT: a den==0 bar no longer deletes - only an actual sub-threshold vote takes an arrow down. This alone ends the wipe half of the flicker even where snapshots are missing (before the first era). 4. Arming moved from an era-counter diff (which fires at era BOUNDARIES, i.e. precisely when caches are about to be wiped) to g_warriorOverlayArmRequest, set by each RankTiersFromOos - "a member's snapshot just got fresher", the only event a redraw can act on. 60s rate limit collapses the four members' burst into one sweep. Classic-only charts arm once at start. 5. Census now reports the direction split - "922 had a voter (610 buy / 312 sell)" - so "the vote leans buy" is checkable from the log instead of inferred from arrow colours. Also visible in the log and worth knowing: the threshold flip-flopped 30 -> 40 -> 30 across the evening's re-inits (census lines at 21:42-21:51 ran at 40), so part of the observed blankness was configuration, not code. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 22:03:20 -04:00
//--- The sweep's data source - see CExpertSignalCustom::SnapshotVoteAt for why this is a
//--- snapshot and not the live cache.
virtual bool SnapshotVoteAt(const int idx, double &signedVote) override
{
signedVote = 0.0;
if(idx < 0 || idx >= m_overlaySnapBars)
return false;
double sig = m_overlaySigSnap[idx];
if(sig == -2.0 || !MathIsValidNumber(sig))
return false;
signedVote = LiveVoteContribution(sig);
return true;
}
feat(chart): reconstruct the filtered view behind the handover point Completes the filtered view from 282b535, which only reached forward of attach. On a multi-hour training run that is the entire time you are looking at the chart, so the answer to "how would the whole bot have traded" was blank exactly when it was wanted. The sweep lives on the AGGREGATE signal, which is the only object holding every filter. AI members contribute their CACHED per-bar decision from the era scan - no inference re-runs, the cache already spans the chart - and the classic ladders are replayed with EvalShift(i), the same mechanism CSignalMETA's candidate sweep uses and exact because every classic pattern condition anchors on StartIndex(). Combination is the live one: weighted mean over voting filters, abstentions out of both sums, against Min_Vote_Open. THE REPLAY CORRUPTS LIVE JOURNALING IF LEFT UNGUARDED, and this is the part that is not obvious. Live journaling reads m_active_pattern_long/short from the PREVIOUS Direction() call. Replaying hundreds of past bars between two live bars leaves those slots holding whichever bar the sweep stopped on, so the next live bar journals that pattern under the current timestamp - a corrupted row in the very table pattern win rates are computed from, which is now also where vote weights come from. Save/RestoreVoteState() brackets every replayed call. CSignalMETA gets away without it only because its sweep runs once, at the first era, before any of that state matters. TWO SOURCES OF TRUTH, KEPT APART. A reconstruction cannot know the broker rejected an order - it has no stops level, ATR warm-up or swing-history sync as they were at that moment - so it is an upper bound: honest about the vote, optimistic about placement. It therefore stops dead at the handover bar, which is latched ONCE so later rebuilds cannot creep it forward and start overwriting real decisions with guesses, and its arrows say "reconstructed (vote only - order validation not replayed)" in the tooltip. Someone comparing two arrows either side of that line has to be able to tell which is a record and which is a replay, and the chart is the only place they look. Re-armed on any era boundary (summed era counters), because that is when the answer changes - RankTiersFromOos has just re-derived every tier's vote weight - and only between sweeps, so a restart cannot leave the previous pass's tail undrawn. Chunked at 150 bars per timer slice: each bar replays Direction() on every classic filter, which is real indicator work on the chart thread, and an unchunked sweep here is the 2026-07-26 arrow-restore freeze waiting to happen. SIGNAL_RESCAN_LOOKBACK_BARS moves to ExpertSignalCustom.mqh alongside SIG_ARROW_PREFIX - same include-order reason, and the two rebuilds should reach the same distance or the raw and filtered views are not comparable. Known gap: on a classic-only chart the reconstruction is built once and not refreshed when the hourly DB ranking moves the classic weights. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 16:05:13 -04:00
virtual bool CachedVoteAt(const int idx, double &signedVote) override
{
signedVote = 0.0;
if(idx < 0 || idx >= ArraySize(m_arrowSignalCache))
return false;
double sig = m_arrowSignalCache[idx];
if(sig == -2.0 || !MathIsValidNumber(sig))
return false;
signedVote = LiveVoteContribution(sig);
return true;
}
//--- buckets the live confidence magnitude into one of the 4 tiers above - see m_pattern_0's
//--- declaration comment. Public so PollTraining()/status-display code could surface which tier is
//--- currently active if ever useful, though LongCondition/ShortCondition are the only callers today.
int ConfidenceTier(void);
fix(gate): the ensemble gate certified a vote the EA never casts g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on the same criterion the live trade does". It did not. Two independent mismatches, both silent: CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100 number straight off the softmax head. Live contributes m_weight x the tier's pattern weight, and BOTH of those are rewritten from the signal DB by UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share an axis and nothing relates them, so the same bar was one number to the gate and a different one to the order path. Same shape as the 2026-08-09 geometry incident: certified on one game, paid on another. DENOMINATOR. The gate divided by the member count, so an abstaining member pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero contribution in BOTH the sum and the count (`if(direction == 0) continue;` before `number++`) - live is a mean over VOTERS. The gate was therefore scoring a strictly more agreement-heavy set of bars than the EA trades. The contribution hook's own comment asserted the opposite ("abstentions dilute the average exactly as they do in the live vote"), while the AI_CHOICE enum 20 lines away correctly documented union semantics. LiveVoteContribution() is now the single definition of "what this member votes", called from the gate; the live path reaches the same arithmetic through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually voted, separately from who evaluated the bar, because those are the divisor and the shared-population test respectively. ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar wrapper - the OOS scan holds the scanned bar's decision in a local, and dPrevSignal is a different bar. NOT changed, deliberately: the gate still does not model live NMS declustering, and the per-member solo gate still scores every directional call rather than threshold-clearing ones. Both are selection-metric changes and this codebase has twice been bitten by switching one blind. Also corrects a stale paragraph in m_pattern_0's declaration block quoting 80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has been since the confidence floor and alternation gate were removed; the block carried both tables at once, and the dead one was quoted back as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
//--- The same bucketing asked of an ARBITRARY decision value rather than of dPrevSignal. Split out
//--- so the OOS scan can ask "what tier would this scanned bar have voted at" - it holds the bar's
//--- decision in a local, and dPrevSignal is the LIVE bar's, which is a different bar entirely.
int ConfidenceTierFor(const double signal);
int PatternWeightForTier(int tier);
//--- THE VOTE THIS MEMBER WOULD CAST, in the units CExpertSignalCustom::Direction() actually
//--- sums: m_weight (0..1, DB-ranked) x the tier's pattern weight (0..100, DB-ranked), signed +
//--- for Buy and - for Sell, and exactly 0.0 when the decision is Neutral (an abstention, which
//--- live drops from BOTH the sum and the divisor).
fix(gate): the ensemble gate certified a vote the EA never casts g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on the same criterion the live trade does". It did not. Two independent mismatches, both silent: CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100 number straight off the softmax head. Live contributes m_weight x the tier's pattern weight, and BOTH of those are rewritten from the signal DB by UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share an axis and nothing relates them, so the same bar was one number to the gate and a different one to the order path. Same shape as the 2026-08-09 geometry incident: certified on one game, paid on another. DENOMINATOR. The gate divided by the member count, so an abstaining member pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero contribution in BOTH the sum and the count (`if(direction == 0) continue;` before `number++`) - live is a mean over VOTERS. The gate was therefore scoring a strictly more agreement-heavy set of bars than the EA trades. The contribution hook's own comment asserted the opposite ("abstentions dilute the average exactly as they do in the live vote"), while the AI_CHOICE enum 20 lines away correctly documented union semantics. LiveVoteContribution() is now the single definition of "what this member votes", called from the gate; the live path reaches the same arithmetic through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually voted, separately from who evaluated the bar, because those are the divisor and the shared-population test respectively. ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar wrapper - the OOS scan holds the scanned bar's decision in a local, and dPrevSignal is a different bar. NOT changed, deliberately: the gate still does not model live NMS declustering, and the per-member solo gate still scores every directional call rather than threshold-clearing ones. Both are selection-metric changes and this codebase has twice been bitten by switching one blind. Also corrects a stale paragraph in m_pattern_0's declaration block quoting 80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has been since the confidence floor and alternation gate were removed; the block carried both tables at once, and the dead one was quoted back as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
double LiveVoteContribution(const double signal);
//--- methods of setting adjustable parameters No public setter for m_initialNeuronsCount. An
//--- external setter could only ever be called after construction and would either be ignored
//--- (if before init) or silently re-key the model mid-run (if after).
void OutputNeuronsCount(int value) { m_outputNeuronsCount = value; }
2026-08-19 18:55:36 -04:00
//--- No setters for m_hiddenLayersCount / m_lstmHiddenSize / m_convFilterCount: the taper's
//--- endpoints are derived, not configured. See BuildFreshTopology()'s taper block.
void MinDirectionalRecall(int value) { m_minDirectionalRecallPct = value; }
//--- MinSignalConfidence(double) removed with the AI entry floor - confidence now reaches the
//--- trade decision as vote weight (ConfidenceTier), gated by the one Min vote to open threshold
//--- that the classic votes already answer to.
feat(ai): logit-adjusted loss, replacing oversampling and the post-hoc prior Menon et al. 2021 (ICLR), "Long-tail learning via logit adjustment": add tau*log(prior_c) to each class logit inside the training gradient. Softmax CE on adjusted logits is consistent for BALANCED error - the metric checkpoint selection already ranks on - so the loss and the deploy decision finally optimize the same thing. The engine already computed a true softmax + categorical-CE gradient and wrote it over the per-neuron sigmoid delta, so this is an offset added to three logits in the two places that gradient is built (backProp scalar path and backPropOCL). No backend, kernel or DLL change; the forward pass and every inference path are untouched, which is the point - the network learns to absorb the offset, so its raw argmax becomes the balanced-optimal decision with nothing applied at inference. Replaces rather than stacks. Minority replay is disabled while this is on, and the post-hoc inference prior is forced off. Stacking is not a theoretical worry: simulated on the measured 1118/1119/34298 distribution in the weak-signal regime, plain CE collapses to Neutral (33.4% balanced, Buy 0%), replay reaches 48.1%, logit adjustment 50.9% with better balance - and BOTH together score 45.4% with Neutral recall at 0%, worse than either alone. Buda et al. 2018 predicts exactly that. Motivation from the six-chart run: every topology took one direction to ~50% recall and abandoned the other, the direction chosen arbitrarily (the batch-norm control went Buy 1% / Sell 42%, the inverse of the other five). One era in 1,301 cleared the per-class recall floor. Fingerprinted conditionally, so the converged 60.7% models on disk keep their filenames and stay loadable as the fallback. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:05:14 -04:00
void LogitAdjustTau(double value) { m_logitAdjustTau = MathMax(0.0, value); }
void FreezePriorCalibration(bool value) { m_freezePriorCalibration = value; }
void SignalClusterWindow(int value) { m_signalClusterWindow = value; }
void SwingConfirmationBars(int value) { m_swingConfirmationBars = value; }
feat(ai): TrainingTarget input - fractal-direction label for the direction models User direction (2026-08-15): back to predicting swing turns, D1 charts, fractals over ZigZag pivots (their call - balances classes, matches the reference library target, and a 5-bar fractal confirms 2 bars after its extreme so labels resolve nearly to the present with no repaint embargo). - TRAINING_TARGET enum + TrainingTarget input: TARGET_BARRIER (Market default - existing models keep their meaning and fingerprints) or TARGET_FRACTAL (private default). - FractalDirectionLabel (Labels.mqh): per-bar 3-class label = direction from the bar close to the next confirmed strict 5-bar fractal extreme, costs charged in the same bid-series convention as the barrier label, Neutral when the move cannot clear max(2 spreads, 0.10 ATR) or on an outside bar (both-extreme bars are unorderable within OHLC). - The barrier walk still runs in full: measured SL/TP geometry, the expectancy scan, excursion caches and the era gate all keep scoring what a trade at the EA's own stop/target actually collected - only the TRAINING label changes. NOT the pre-b4a704d "is this bar the pivot" form; that target's 31:1 imbalance stays retired. - Fingerprint token |TGT:FRA1 so switching targets trains a separate model; AI_META unaffected (guarded setter). - Private defaults: AIType back to AI_HYBRID (direction topology needed) + TrainingTarget=TARGET_FRACTAL = drop-on-D1-chart workflow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 04:44:10 -04:00
//--- Called from ConfigureAISignal when the TrainingTarget input selects the fractal label. Guarded
//--- so CSignalMETA (whose constructor already claimed target 1) can never be flipped: the meta
//--- head's 2-output topology and candidate pipeline are incompatible with a per-bar 3-class label.
void TrainTargetFractal(void) { if(m_trainTarget == 0) m_trainTarget = 2; }
//--- ENSEMBLE MEMBERSHIP (two or more direction NNs enabled on one chart).
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
void EnsembleMember(bool value, double voteThreshold = -1.0)
{
m_ensembleMember = value;
if(voteThreshold >= 0.0)
g_ensembleVoteThreshold = voteThreshold;
if(value && m_ensembleIndex < 0)
{
int n = ArraySize(g_warriorEnsemble);
ArrayResize(g_warriorEnsemble, n + 1);
g_warriorEnsemble[n] = GetPointer(this);
m_ensembleIndex = n;
}
}
//--- Minimum era among the ensemble members still genuinely training. Falls back to this
//--- member's own era when nothing qualifies, which makes the barrier a no-op rather than a
//--- lock.
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
long EnsembleMinTrainingEra(void)
{
long minEra = LONG_MAX;
for(int i = 0; i < ArraySize(g_warriorEnsemble); i++)
{
CExpertSignalAIBase *mm = g_warriorEnsemble[i];
if(CheckPointer(mm) == POINTER_INVALID)
continue;
if(mm.m_trainingComplete || mm.m_trainingStopRequested || mm.m_trainingPaused || !mm.m_isInitialized)
continue;
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
//--- THE LIVENESS EXEMPTION (see ENSEMBLE_BARRIER_STUCK_MS). The three flags above are all
//--- VOLUNTARY - a member that chose to stop participating.
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
if(mm.m_barrierExcluded)
continue;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
if(mm.m_eraCount < minEra)
minEra = mm.m_eraCount;
}
return (minEra == LONG_MAX) ? m_eraCount : minEra;
}
fix(ensemble): the era barrier read healthy startup work as a dead member Reported symptom: one member at era 17 while the rest sat at era 2, with the combined vote never scoring. Two faults compound to produce exactly that, and neither needs a broken model to trigger. FIRST - BUSY WAS READ AS STUCK. BarrierEraHeartbeat() decides liveness from one signal: has m_eraCount changed in the last 12 minutes. But Train() returns early, before the era loop, for three ONE-TIME phases that never touch m_eraCount - the label-cache prebuild, the pattern-DB backfill and the OOS simulation walk - and those are precisely what a slow topology spends its first many minutes doing. A member grinding steadily through a prebuild therefore looked identical to a dead one and was dropped from the barrier at startup, before it had trained a single era. The constant's own comment states the flawed premise: "comfortably past the slowest healthy ERA on the deepest chart" - true, and not the question being asked. Those three branches now call NoteBarrierProgress() and a chunk of phase work re-arms the watchdog exactly as an era does. SECOND - EXCLUSION HAD NO BOUND. Once dropped, a member is skipped by EnsembleMinTrainingEra(). Drop every OTHER member and that loop finds nothing to take a minimum over, falls through to its `return m_eraCount` fallback - the CALLER'S own era - and EnsembleEraBarrierHolds() evaluates `era > era`, false, for everybody. The barrier silently becomes a no-op and the fastest member runs away unbounded. EnsembleMinEraAnyMember() now measures against every still-training member, excluded or not, and a member may lead it by at most ENSEMBLE_MAX_ERA_LEAD eras. The cap is deliberately a real stop rather than a warning. A desynchronised ensemble is not a degraded one: the combined-vote score and the joint checkpoint both require every member on the same era, so weights trained past the cap can never be certified by any gate. The hold reports which of the two it is, because the operator's next move differs - an ordinary barrier hold resolves itself, a lead-cap hold names a member that needs diagnosing and will not resolve on its own. Not yet explained: "only one NN listened to the stop command". The panel now dispatches down the filter tree and reports the count it reached ("training stopped (N model(s))"), so the next run answers that definitively instead of leaving it to inference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 09:35:25 -04:00
//--- Minimum era among still-training members, IGNORING the barrier exclusion. This is what the lead
//--- cap measures against, so an excluded member still BOUNDS the ensemble even though it no longer
//--- BLOCKS it - which is the difference between a liveness escape and an unbounded desync.
long EnsembleMinEraAnyMember(void)
{
long minEra = LONG_MAX;
for(int i = 0; i < ArraySize(g_warriorEnsemble); i++)
{
CExpertSignalAIBase *mm = g_warriorEnsemble[i];
if(CheckPointer(mm) == POINTER_INVALID)
continue;
if(mm.m_trainingComplete || mm.m_trainingStopRequested || mm.m_trainingPaused || !mm.m_isInitialized)
continue;
if(mm.m_eraCount < minEra)
minEra = mm.m_eraCount;
}
return (minEra == LONG_MAX) ? m_eraCount : minEra;
}
//--- How many members are actively consuming training chunks right now: still training AND at
//--- the barrier's minimum era (a member held ABOVE the min declines its calls, so it costs
//--- nothing).
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
int EnsembleActiveTrainers(void)
{
if(!m_ensembleMember)
return 1;
long minEra = EnsembleMinTrainingEra();
int active = 0;
for(int i = 0; i < ArraySize(g_warriorEnsemble); i++)
{
CExpertSignalAIBase *mm = g_warriorEnsemble[i];
if(CheckPointer(mm) == POINTER_INVALID)
continue;
if(mm.m_trainingComplete || mm.m_trainingStopRequested || mm.m_trainingPaused || !mm.m_isInitialized)
continue;
if(mm.m_eraCount <= minEra)
active++;
}
return MathMax(active, 1);
}
//--- True when this member has finished more eras than the slowest still-training member and must
//--- wait at the era barrier - checked at Train()'s entry (see the barrier note there).
bool EnsembleEraBarrierHolds(void)
{
if(!m_ensembleMember || m_trainingComplete)
return false;
fix(ensemble): the era barrier read healthy startup work as a dead member Reported symptom: one member at era 17 while the rest sat at era 2, with the combined vote never scoring. Two faults compound to produce exactly that, and neither needs a broken model to trigger. FIRST - BUSY WAS READ AS STUCK. BarrierEraHeartbeat() decides liveness from one signal: has m_eraCount changed in the last 12 minutes. But Train() returns early, before the era loop, for three ONE-TIME phases that never touch m_eraCount - the label-cache prebuild, the pattern-DB backfill and the OOS simulation walk - and those are precisely what a slow topology spends its first many minutes doing. A member grinding steadily through a prebuild therefore looked identical to a dead one and was dropped from the barrier at startup, before it had trained a single era. The constant's own comment states the flawed premise: "comfortably past the slowest healthy ERA on the deepest chart" - true, and not the question being asked. Those three branches now call NoteBarrierProgress() and a chunk of phase work re-arms the watchdog exactly as an era does. SECOND - EXCLUSION HAD NO BOUND. Once dropped, a member is skipped by EnsembleMinTrainingEra(). Drop every OTHER member and that loop finds nothing to take a minimum over, falls through to its `return m_eraCount` fallback - the CALLER'S own era - and EnsembleEraBarrierHolds() evaluates `era > era`, false, for everybody. The barrier silently becomes a no-op and the fastest member runs away unbounded. EnsembleMinEraAnyMember() now measures against every still-training member, excluded or not, and a member may lead it by at most ENSEMBLE_MAX_ERA_LEAD eras. The cap is deliberately a real stop rather than a warning. A desynchronised ensemble is not a degraded one: the combined-vote score and the joint checkpoint both require every member on the same era, so weights trained past the cap can never be certified by any gate. The hold reports which of the two it is, because the operator's next move differs - an ordinary barrier hold resolves itself, a lead-cap hold names a member that needs diagnosing and will not resolve on its own. Not yet explained: "only one NN listened to the stop command". The panel now dispatches down the filter tree and reports the count it reached ("training stopped (N model(s))"), so the next run answers that definitively instead of leaving it to inference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 09:35:25 -04:00
//--- The ordinary barrier: ahead of the slowest member that is still in it.
if(m_eraCount > EnsembleMinTrainingEra())
return true;
//--- ...and the backstop for a member that has been EXCLUDED from that minimum. Without this the
//--- exclusion is a licence to run away without limit - see ENSEMBLE_MAX_ERA_LEAD.
return (m_eraCount - EnsembleMinEraAnyMember() >= ENSEMBLE_MAX_ERA_LEAD);
}
//--- True when the hold above is the LEAD CAP rather than the ordinary barrier, i.e. we are waiting on
//--- a member the barrier has already given up on. Reported differently because the operator's next
//--- move differs: an ordinary hold resolves itself, this one needs the named member diagnosed.
bool EnsembleLeadCapHolds(void)
{
return (m_ensembleMember && !m_trainingComplete &&
m_eraCount <= EnsembleMinTrainingEra() &&
m_eraCount - EnsembleMinEraAnyMember() >= ENSEMBLE_MAX_ERA_LEAD);
}
//--- A long one-time phase advanced a chunk. Called from the prebuild / backfill / simulation
//--- branches of Train(), which all return before the era loop and so leave m_eraCount untouched
//--- for as long as the phase lasts.
fix(ensemble): the era barrier read healthy startup work as a dead member Reported symptom: one member at era 17 while the rest sat at era 2, with the combined vote never scoring. Two faults compound to produce exactly that, and neither needs a broken model to trigger. FIRST - BUSY WAS READ AS STUCK. BarrierEraHeartbeat() decides liveness from one signal: has m_eraCount changed in the last 12 minutes. But Train() returns early, before the era loop, for three ONE-TIME phases that never touch m_eraCount - the label-cache prebuild, the pattern-DB backfill and the OOS simulation walk - and those are precisely what a slow topology spends its first many minutes doing. A member grinding steadily through a prebuild therefore looked identical to a dead one and was dropped from the barrier at startup, before it had trained a single era. The constant's own comment states the flawed premise: "comfortably past the slowest healthy ERA on the deepest chart" - true, and not the question being asked. Those three branches now call NoteBarrierProgress() and a chunk of phase work re-arms the watchdog exactly as an era does. SECOND - EXCLUSION HAD NO BOUND. Once dropped, a member is skipped by EnsembleMinTrainingEra(). Drop every OTHER member and that loop finds nothing to take a minimum over, falls through to its `return m_eraCount` fallback - the CALLER'S own era - and EnsembleEraBarrierHolds() evaluates `era > era`, false, for everybody. The barrier silently becomes a no-op and the fastest member runs away unbounded. EnsembleMinEraAnyMember() now measures against every still-training member, excluded or not, and a member may lead it by at most ENSEMBLE_MAX_ERA_LEAD eras. The cap is deliberately a real stop rather than a warning. A desynchronised ensemble is not a degraded one: the combined-vote score and the joint checkpoint both require every member on the same era, so weights trained past the cap can never be certified by any gate. The hold reports which of the two it is, because the operator's next move differs - an ordinary barrier hold resolves itself, a lead-cap hold names a member that needs diagnosing and will not resolve on its own. Not yet explained: "only one NN listened to the stop command". The panel now dispatches down the filter tree and reports the count it reached ("training stopped (N model(s))"), so the next run answers that definitively instead of leaving it to inference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 09:35:25 -04:00
void NoteBarrierProgress(void)
{
m_barrierPhaseProgress = true;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
}
//--- Era-advance watchdog for the barrier, kept SEPARATE from m_lastEraCompleteTick on purpose.
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
void BarrierEraHeartbeat(void)
{
if(!m_ensembleMember)
return;
uint nowTick = GetTickCount();
if(m_barrierEraTick == 0 || m_barrierEraSeen != m_eraCount)
{
//--- Progress (or the first observation). Rejoining is unconditional and immediate: a member
//--- that just completed an era is by definition not stuck, whatever it was doing before.
if(m_barrierExcluded)
Print(ID + ": REJOINING THE ERA BARRIER at era " + IntegerToString((int)m_eraCount) +
" - it completed an era, so it is training again. It is behind the rest of the"
" ensemble, which means it now sets the minimum and the others wait for it to catch"
" up. The combined-vote score resumes once every member reports the same era.");
m_barrierExcluded = false;
m_barrierEraSeen = m_eraCount;
m_barrierEraTick = nowTick;
return;
}
fix(ensemble): the era barrier read healthy startup work as a dead member Reported symptom: one member at era 17 while the rest sat at era 2, with the combined vote never scoring. Two faults compound to produce exactly that, and neither needs a broken model to trigger. FIRST - BUSY WAS READ AS STUCK. BarrierEraHeartbeat() decides liveness from one signal: has m_eraCount changed in the last 12 minutes. But Train() returns early, before the era loop, for three ONE-TIME phases that never touch m_eraCount - the label-cache prebuild, the pattern-DB backfill and the OOS simulation walk - and those are precisely what a slow topology spends its first many minutes doing. A member grinding steadily through a prebuild therefore looked identical to a dead one and was dropped from the barrier at startup, before it had trained a single era. The constant's own comment states the flawed premise: "comfortably past the slowest healthy ERA on the deepest chart" - true, and not the question being asked. Those three branches now call NoteBarrierProgress() and a chunk of phase work re-arms the watchdog exactly as an era does. SECOND - EXCLUSION HAD NO BOUND. Once dropped, a member is skipped by EnsembleMinTrainingEra(). Drop every OTHER member and that loop finds nothing to take a minimum over, falls through to its `return m_eraCount` fallback - the CALLER'S own era - and EnsembleEraBarrierHolds() evaluates `era > era`, false, for everybody. The barrier silently becomes a no-op and the fastest member runs away unbounded. EnsembleMinEraAnyMember() now measures against every still-training member, excluded or not, and a member may lead it by at most ENSEMBLE_MAX_ERA_LEAD eras. The cap is deliberately a real stop rather than a warning. A desynchronised ensemble is not a degraded one: the combined-vote score and the joint checkpoint both require every member on the same era, so weights trained past the cap can never be certified by any gate. The hold reports which of the two it is, because the operator's next move differs - an ordinary barrier hold resolves itself, a lead-cap hold names a member that needs diagnosing and will not resolve on its own. Not yet explained: "only one NN listened to the stop command". The panel now dispatches down the filter tree and reports the count it reached ("training stopped (N model(s))"), so the next run answers that definitively instead of leaving it to inference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 09:35:25 -04:00
//--- BUSY IS NOT STUCK. A member grinding through the label prebuild, the DB backfill or the OOS
//--- simulation walk never touches m_eraCount, so the era test above cannot see it working. Any
//--- chunk of those phases counts as progress and re-arms the clock, exactly as an era does.
if(m_barrierPhaseProgress)
{
m_barrierPhaseProgress = false;
if(m_barrierExcluded)
Print(ID + ": REJOINING THE ERA BARRIER - it is still on era " + IntegerToString((int)m_eraCount) +
" but is making progress through a one-time preparation phase, not stuck.");
m_barrierExcluded = false;
m_barrierEraTick = nowTick;
return;
}
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with four of their eight members completely silent. Nothing in this commit guesses at why the sweep fails - the last five guesses were all wrong. It makes the failure say what it is, and stops one broken member taking its whole chart down with it. WHAT THE LOG ACTUALLY SAYS, before any of this. - The running build IS d9f834d (pulled 14:18, compiled 14:19:01, 0 errors), so every depth instrument from 1dda479/7e63a8b/45c9e21 was live. - It printed NOTHING. Zero "PRIMING", zero "CAPPED", zero "Per-indicator depth" in 27 MB of journal. The instrument built to find the depth shortfall returned "not this". - On USDJPY at 14:24, CONV-cad8 completed eras 0 AND 1 across all 50,179 bars - same chart, same 832-value window, same indicators, byte-identical fingerprint - while LSTM-cad8 and HYB-cad8 reported ok=0 failed=50163. So it is not the symbol, the history, the bar count or the indicator depth. It is per-member. - ok=0 means the NEWEST anchors failed too, and a short indicator cannot do that. The depth reading in project_silent_block_failures is therefore retired by its own instrumentation. THE ROOT CAUSE IS STILL UNKNOWN and this commit does not claim one. 1. THE DEPTH GATE'S SILENT PATH WAS THE STATE IT WAS HUNTING. ServableBars() read `if(servable <= 0 || servable >= want) return want;` - one branch over three unrelated states, silent in all of them: enabled == 0 -> nothing tunable is on. No cap. Healthy. enabled > 0, servable == -1 -> a handle answered INVALID. enabled > 0, servable == 0 -> created, never calculated. BarsCalculated() returns -1 for a dead handle, so a dead MA is indistinguishable from "no tunable indicators enabled" - and both returned `want` without printing a character. That is exactly the state a per-member, every-index, depth-independent failure produces, and it is the single reason a build carrying full depth instrumentation logged nothing through the whole outage. TunableBarsCalculated() now also reports HOW MANY indicators it consulted, and the dead-handle case is reported (latched, with per-handle depths). The RETURN is deliberately unchanged - what to do about a dead handle is not yet known, and changing control flow on an unproven cause is how the last four fixes here went wrong. SettledBars() routes its three pass-through states via ServableBars() so the report is reachable from the training sweep, which is the only caller that hits it. 2. THE STALL REPORT NAMED A SLOT, NEVER A BLOCK. "lookback slot 0 REJECTED (window had 24 of 832 values)" plus a guess ("an indicator warm-up or a history-edge read"). Which guard fired was INFERRED by counting 4+5+4+4+6+1 = 24 and concluding feature 25 must be the MA. The arithmetic was right; every conclusion drawn from it was wrong, because a value count names a POSITION and a position cannot tell cold from capped from invalid from off-the-end. Every guard that can reject a bar now records itself - m_featureFailBlock - and the report carries it, the series index, IndicatorDepthReport()'s per-handle depths, and for each indicator whether the NEWEST bar reads. That last field is the whole diagnosis in one word: newest-also-EMPTY means the buffer is unreadable everywhere (cold or dead handle), newest-reads means a genuine history edge. Instrumented: open, ATR, MA, RSI, MACD, Ichimoku, and all five AD blocks via ADIndicatorCold(). 3. THE TOTAL-FAILURE BACKOFF WAS GATED ON THE WRONG QUESTION. It armed only when m_featureFailTransient was set. Keeping that flag correct across every guard is a list that has to stay right forever - the same shape of fix the feature cache abandoned for the same reason - and the gate is pointless anyway: a sweep where ZERO of 50,163 bars produced a window will produce zero again if it restarts a millisecond later, transient or not. Doing that at full speed is what starved six indicator threads on a six-core box. The backoff is now unconditional on a total failure. The flag keeps its real job, deciding whether a MISS may be cached, which is a per-bar question and not a scheduling one. 4. THE ERA BARRIER DEADLOCKED, AND SILENCED THE MEMBERS IT FROZE. EnsembleMinTrainingEra() exempted deployed, stopped and paused members and its comment concluded "so nothing deadlocks". Those three are all VOLUNTARY. A member that simply CANNOT finish an era is none of them, so it pinned the minimum at its own era with no time limit - and the hold branch's only action was `m_lastEraCompleteTick = GetTickCount()`, which silences the stall watchdog. So on USDJPY the two members that could not train reported, and the two healthy members frozen behind them wrote nothing anywhere. The outage was visible only through the members that were not suffering it. - BarrierEraHeartbeat() stamps a clock on real era CHANGE, kept separate from m_lastEraCompleteTick precisely because the barrier resets that one. Only a member AT the minimum can be a blocker; a member ahead is idle by design and is never counted as stuck. - After ENSEMBLE_BARRIER_STUCK_MS (12 min) a non-advancing member is dropped from the barrier minimum. It keeps training and rejoins the instant it completes an era - at which point, being behind, it legitimately becomes the minimum again, which is the documented resumed-laggard behaviour. - Both transitions say so loudly, and the release states plainly that the combined-vote score cannot be computed while the ensemble is desynchronised. - A held member now writes a rate-limited journal line naming WHICH members it is waiting on, so the blocker is read off one line. 5. THE PANEL FLICKER. OnTickHandler gates its terse writer on !m_trainRunActive, and a barrier-held member returns from Train() before ever setting it - so both writers thought they were the only one updating the label and fought every tick. That is the reported "Getting ready..." <-> "Waiting at era N for slower ensemble members" oscillation, and it hit Perceptron but not Convolutional purely because Convolutional had a run active from a completed era and Perceptron, resumed from disk, never did. Train()'s message is the specific one, so it wins. NEXT STEP once this is running: the stall line now ends in "REJECTED BY: ..." and the per-handle depths. Read it. Do not reason around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:21:15 -04:00
//--- Same era as last look. Only a member that is AT the minimum can be the one blocking: a member
//--- ahead of it is not advancing because the barrier is holding it, which is correct behaviour and
//--- must never be mistaken for being stuck.
if(m_barrierExcluded || m_eraCount > EnsembleMinTrainingEra())
return;
if(nowTick - m_barrierEraTick < ENSEMBLE_BARRIER_STUCK_MS)
return;
m_barrierExcluded = true;
PrintFormat("%s: RELEASING THE ERA BARRIER - this member has not completed an era in %.0f minutes"
" (still at era %d) and every other member on this chart has been waiting on it for"
" that entire time. It is excluded from the barrier minimum so the rest can advance;"
" it keeps training and rejoins the moment it finishes an era. READ THE TRAIN STALL"
" LINE ABOVE for why it is not finishing - the barrier only reports that it is stuck,"
" never why. NOTE: while the ensemble is desynchronised the combined-vote OOS score"
" cannot be computed (it scores only bars EVERY member contributed at the same era),"
" so no ensemble verdict will be published until this member catches up.",
ID, (nowTick - m_barrierEraTick) / 60000.0, (int)m_eraCount);
}
//--- Pass-3 hook: record the VOTE this member would have cast on this OOS bar into the combined-
//--- vote buffer. signedVote is in live vote units - m_weight x tier pattern weight, signed by
//--- direction (see LiveVoteContribution()) - NOT the raw confidence this used to carry.
fix(gate): the ensemble gate certified a vote the EA never casts g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on the same criterion the live trade does". It did not. Two independent mismatches, both silent: CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100 number straight off the softmax head. Live contributes m_weight x the tier's pattern weight, and BOTH of those are rewritten from the signal DB by UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share an axis and nothing relates them, so the same bar was one number to the gate and a different one to the order path. Same shape as the 2026-08-09 geometry incident: certified on one game, paid on another. DENOMINATOR. The gate divided by the member count, so an abstaining member pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero contribution in BOTH the sum and the count (`if(direction == 0) continue;` before `number++`) - live is a mean over VOTERS. The gate was therefore scoring a strictly more agreement-heavy set of bars than the EA trades. The contribution hook's own comment asserted the opposite ("abstentions dilute the average exactly as they do in the live vote"), while the AI_CHOICE enum 20 lines away correctly documented union semantics. LiveVoteContribution() is now the single definition of "what this member votes", called from the gate; the live path reaches the same arithmetic through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually voted, separately from who evaluated the bar, because those are the divisor and the shared-population test respectively. ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar wrapper - the OOS scan holds the scanned bar's decision in a local, and dPrevSignal is a different bar. NOT changed, deliberately: the gate still does not model live NMS declustering, and the per-member solo gate still scores every directional call rather than threshold-clearing ones. Both are selection-metric changes and this codebase has twice been bitten by switching one blind. Also corrects a stale paragraph in m_pattern_0's declaration block quoting 80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has been since the confidence floor and alternation gate were removed; the block carried both tables at once, and the dead one was quoted back as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
void EnsembleOosContribute(const int barIdx, const double signedVote,
feat(vote): thresholds become confidence percentages, on ONE scale everywhere User request: "the entry/exit thresholds are manual numbers, I would like them to be confidence percentages, so the current 20 would be only 20% confidence in a profitable trade." WHY 20 WAS EVER SENSIBLE. Under UseDatabaseRanking both factors of a filter's contribution are win rates: the pattern weight is that pattern's measured win rate (UpdateSignalsWeights -> ApplyPatternWeight) and m_weight is the filter's average win rate over its patterns, /100. Dividing the sum by the VOTER COUNT therefore produced a mean of PRODUCTS of two win rates - a genuinely 60%-accurate filter firing a 60% pattern scored 0.60 x 60 = 36. The number was never on a probability scale, so its magnitude meant nothing on its own. Dividing by Sum(m_weight) instead makes it a weighted MEAN of win rates, which is a win rate: result = Sum(w_i*p_i)/Sum(w_i). Every voter at 60% now reads 60; MACD's double-divergence pattern (weight 100) voting alone reads 100. m_weight stops being a discount on the probability and becomes how much a filter's opinion COUNTS - which is what a module weight should always have been. Default Min_Vote_Open 20 -> 50: not a tightening, the same bar re-expressed. ONE SCALE, EVERYWHERE - the part that made this bigger than a rescale. Three other places compared against a 0..1 softmax confidence and would each have become a fresh currency mismatch the moment the input changed meaning: * the AI early-exit route (LiveSignedConfidence vs m_ai_exit_threshold) now reads m_lastAiVote - the AI filters' own weighted mean, undiluted by the classic side, which is the only reason that route exists - against the same m_threshold_close the averaged vote uses. m_ai_exit_threshold is retired rather than left dangling. * m_oosDecisionSeries now carries the vote, not the confidence, so the exit SIMULATION stops modelling a close rule the EA does not run. * ExitPolicy() clamped anything > 1.0 to zero. Passing the unscaled input through that would have silently switched vote exits off in the simulation while live went on running them - found before it shipped; the bound now tracks the scale. LiveSignedConfidence() is deliberately untouched and still 0..1: MM sizing, SL/TP scaling and the intelligent trailing want a model confidence, not a win rate. CALIBRATION CAVEAT, stated in the code where the claim is made: this is only a real probability to the extent the pattern weights are. A pattern with fewer than MIN_TRADES_FOR_WIN_RATE journaled trades keeps its DEFAULT weight - a designed prior (25/50/75/100 for the AI tiers), not a measurement. Until the signal DB fills, "60" means "the designed conviction of the patterns that fired". Closing that gap is the next commit. Also corrects VOTE_CLOSE_PRESETS' comment, which documented the two scales this removes. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:52:08 -04:00
const double voteWeight,
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
const bool winLong, const bool winShort, const bool dirLabel)
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
{
if(!m_ensembleMember || m_ensembleIndex < 0 || m_ensembleIndex >= 8)
return;
datetime t = m_Time.GetData(barIdx);
if(t <= 0)
return;
if(g_ensVoteEra != m_eraCount)
{
//--- first contribution of a new era resets the buffer (the era barrier keeps members aligned,
//--- so a mismatched stamp means "previous era's rows", never "a sibling's different era")
g_ensVoteEra = m_eraCount;
g_ensVoteRows = 0;
g_ensVoteDoneMask = 0;
ArrayInitialize(g_ensVoteCursor, 0);
}
int bit = (1 << m_ensembleIndex);
//--- monotonic cursor first (members scan bars oldest-to-newest, so the match is O(1) amortized),
//--- full wrap-around only when per-member window failures desynchronize the sequences
int row = -1;
int start = g_ensVoteCursor[m_ensembleIndex];
if(start > g_ensVoteRows)
start = 0;
for(int i = start; i < g_ensVoteRows; i++)
if(g_ensVoteTime[i] == t) { row = i; break; }
if(row < 0)
for(int i = 0; i < start; i++)
if(g_ensVoteTime[i] == t) { row = i; break; }
if(row < 0)
{
if(g_ensVoteRows >= ArraySize(g_ensVoteTime))
{
int cap = g_ensVoteRows + g_ensVoteRows / 2 + 512;
ArrayResize(g_ensVoteTime, cap);
ArrayResize(g_ensVoteSum, cap);
feat(baselines): combining-weight fit, MLP cross-validation, all-lags correlation Three ALGLIB additions, all measurement-only and all under the existing Run_Alglib_Baselines switch. MinBLEIC COMBINING WEIGHTS. The live ensemble weights each member by its own pooled holdout win rate - a defensible prior, but not a fit, and nothing has ever asked what mixture minimises error on the bars the members disagreed about. Two individually-mediocre members wrong in different places can beat one individually better, and a per-member win rate cannot express that because it never looks at them jointly. Solved on the simplex (w >= 0, sum w = 1), which is exactly what MinBLEIC is for. Non-negative because a negative weight asserts "trade the opposite of this member", a claim ~60 effective observations cannot support. Least squares on the signed outcome rather than precision: precision is a STEP function of the threshold that no gradient method can walk, and optimising a smooth proxy for a step decision is how c3daded put every operating point 14pp underwater - so the result is reported in BOTH currencies, the SSE it minimised and the directional hit rate the mixture would actually have scored against the equal mix. If the second does not improve, the first is noise. This needed data that did not exist: g_ensVoteSum accumulates member contributions and the sum destroys the decomposition, while g_ensVoteVoterMask records only WHETHER a member voted, never what. g_ensVoteMember[] keeps them unsummed. The live arithmetic is untouched. ENS_MAX_MEMBERS is 8 and deliberately larger than MAX_AI_SIGNALS (5): independent caps, over-allocating is free, and matching them would make this array silently short the day the registry grows - a cap that has already dropped a member once without saying so. MLPKFoldCVLBFGS. Every baseline row carries a binomial SE, which is the sampling error of SCORING a fixed model and says nothing about how much the FIT moves. One LBFGS run from one random start can land anywhere, and a baseline that cleared or missed the bar on luck of initialisation reads exactly like one that did it on merit. 3 folds, because each is a full retrain. LBFGS not LM - LM builds a Hessian over ~7,700 weights. CCorr ALL-LAGS PROFILE. Added BESIDE the MI lag profile, not instead: MI catches nonlinear dependence and is the stronger negative, which is why it settled the verdict - but its per-lag permutation null limits it to ~20 lags. FFT correlation gets every lag in one O(n log n) pass, so linear structure parked at lag 300 would surface for free. Different question, not a replacement. Walks CONTIGUOUS bars, unlike everything else in this file, because a lag index is meaningless otherwise; both series are mean-centred first since CorrR1D is a raw sum of products; and the max over columns x lags is judged against a Sidak family of exactly that size, not a bare 2-sigma line. fasttransforms.mqh needed its own include - verified that none of ap/optimization/statistics/solvers/linalg reaches it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 07:44:36 -04:00
ArrayResize(g_ensVoteMember, cap * ENS_MAX_MEMBERS);
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
ArrayResize(g_ensVoteMask, cap);
fix(gate): the ensemble gate certified a vote the EA never casts g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on the same criterion the live trade does". It did not. Two independent mismatches, both silent: CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100 number straight off the softmax head. Live contributes m_weight x the tier's pattern weight, and BOTH of those are rewritten from the signal DB by UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share an axis and nothing relates them, so the same bar was one number to the gate and a different one to the order path. Same shape as the 2026-08-09 geometry incident: certified on one game, paid on another. DENOMINATOR. The gate divided by the member count, so an abstaining member pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero contribution in BOTH the sum and the count (`if(direction == 0) continue;` before `number++`) - live is a mean over VOTERS. The gate was therefore scoring a strictly more agreement-heavy set of bars than the EA trades. The contribution hook's own comment asserted the opposite ("abstentions dilute the average exactly as they do in the live vote"), while the AI_CHOICE enum 20 lines away correctly documented union semantics. LiveVoteContribution() is now the single definition of "what this member votes", called from the gate; the live path reaches the same arithmetic through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually voted, separately from who evaluated the bar, because those are the divisor and the shared-population test respectively. ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar wrapper - the OOS scan holds the scanned bar's decision in a local, and dPrevSignal is a different bar. NOT changed, deliberately: the gate still does not model live NMS declustering, and the per-member solo gate still scores every directional call rather than threshold-clearing ones. Both are selection-metric changes and this codebase has twice been bitten by switching one blind. Also corrects a stale paragraph in m_pattern_0's declaration block quoting 80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has been since the confidence floor and alternation gate were removed; the block carried both tables at once, and the dead one was quoted back as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
ArrayResize(g_ensVoteVoterMask, cap);
feat(vote): thresholds become confidence percentages, on ONE scale everywhere User request: "the entry/exit thresholds are manual numbers, I would like them to be confidence percentages, so the current 20 would be only 20% confidence in a profitable trade." WHY 20 WAS EVER SENSIBLE. Under UseDatabaseRanking both factors of a filter's contribution are win rates: the pattern weight is that pattern's measured win rate (UpdateSignalsWeights -> ApplyPatternWeight) and m_weight is the filter's average win rate over its patterns, /100. Dividing the sum by the VOTER COUNT therefore produced a mean of PRODUCTS of two win rates - a genuinely 60%-accurate filter firing a 60% pattern scored 0.60 x 60 = 36. The number was never on a probability scale, so its magnitude meant nothing on its own. Dividing by Sum(m_weight) instead makes it a weighted MEAN of win rates, which is a win rate: result = Sum(w_i*p_i)/Sum(w_i). Every voter at 60% now reads 60; MACD's double-divergence pattern (weight 100) voting alone reads 100. m_weight stops being a discount on the probability and becomes how much a filter's opinion COUNTS - which is what a module weight should always have been. Default Min_Vote_Open 20 -> 50: not a tightening, the same bar re-expressed. ONE SCALE, EVERYWHERE - the part that made this bigger than a rescale. Three other places compared against a 0..1 softmax confidence and would each have become a fresh currency mismatch the moment the input changed meaning: * the AI early-exit route (LiveSignedConfidence vs m_ai_exit_threshold) now reads m_lastAiVote - the AI filters' own weighted mean, undiluted by the classic side, which is the only reason that route exists - against the same m_threshold_close the averaged vote uses. m_ai_exit_threshold is retired rather than left dangling. * m_oosDecisionSeries now carries the vote, not the confidence, so the exit SIMULATION stops modelling a close rule the EA does not run. * ExitPolicy() clamped anything > 1.0 to zero. Passing the unscaled input through that would have silently switched vote exits off in the simulation while live went on running them - found before it shipped; the bound now tracks the scale. LiveSignedConfidence() is deliberately untouched and still 0..1: MM sizing, SL/TP scaling and the intelligent trailing want a model confidence, not a win rate. CALIBRATION CAVEAT, stated in the code where the claim is made: this is only a real probability to the extent the pattern weights are. A pattern with fewer than MIN_TRADES_FOR_WIN_RATE journaled trades keeps its DEFAULT weight - a designed prior (25/50/75/100 for the AI tiers), not a measurement. Until the signal DB fills, "60" means "the designed conviction of the patterns that fired". Closing that gap is the next commit. Also corrects VOTE_CLOSE_PRESETS' comment, which documented the two scales this removes. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:52:08 -04:00
ArrayResize(g_ensVoteWeightSum, cap);
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
ArrayResize(g_ensVoteWinLong, cap);
ArrayResize(g_ensVoteWinShort, cap);
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
ArrayResize(g_ensVoteDirLabel, cap);
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
}
row = g_ensVoteRows++;
g_ensVoteTime[row] = t;
g_ensVoteSum[row] = 0.0;
feat(baselines): combining-weight fit, MLP cross-validation, all-lags correlation Three ALGLIB additions, all measurement-only and all under the existing Run_Alglib_Baselines switch. MinBLEIC COMBINING WEIGHTS. The live ensemble weights each member by its own pooled holdout win rate - a defensible prior, but not a fit, and nothing has ever asked what mixture minimises error on the bars the members disagreed about. Two individually-mediocre members wrong in different places can beat one individually better, and a per-member win rate cannot express that because it never looks at them jointly. Solved on the simplex (w >= 0, sum w = 1), which is exactly what MinBLEIC is for. Non-negative because a negative weight asserts "trade the opposite of this member", a claim ~60 effective observations cannot support. Least squares on the signed outcome rather than precision: precision is a STEP function of the threshold that no gradient method can walk, and optimising a smooth proxy for a step decision is how c3daded put every operating point 14pp underwater - so the result is reported in BOTH currencies, the SSE it minimised and the directional hit rate the mixture would actually have scored against the equal mix. If the second does not improve, the first is noise. This needed data that did not exist: g_ensVoteSum accumulates member contributions and the sum destroys the decomposition, while g_ensVoteVoterMask records only WHETHER a member voted, never what. g_ensVoteMember[] keeps them unsummed. The live arithmetic is untouched. ENS_MAX_MEMBERS is 8 and deliberately larger than MAX_AI_SIGNALS (5): independent caps, over-allocating is free, and matching them would make this array silently short the day the registry grows - a cap that has already dropped a member once without saying so. MLPKFoldCVLBFGS. Every baseline row carries a binomial SE, which is the sampling error of SCORING a fixed model and says nothing about how much the FIT moves. One LBFGS run from one random start can land anywhere, and a baseline that cleared or missed the bar on luck of initialisation reads exactly like one that did it on merit. 3 folds, because each is a full retrain. LBFGS not LM - LM builds a Hessian over ~7,700 weights. CCorr ALL-LAGS PROFILE. Added BESIDE the MI lag profile, not instead: MI catches nonlinear dependence and is the stronger negative, which is why it settled the verdict - but its per-lag permutation null limits it to ~20 lags. FFT correlation gets every lag in one O(n log n) pass, so linear structure parked at lag 300 would surface for free. Different question, not a replacement. Walks CONTIGUOUS bars, unlike everything else in this file, because a lag index is meaningless otherwise; both series are mean-centred first since CorrR1D is a raw sum of products; and the max over columns x lags is judged against a Sidak family of exactly that size, not a bare 2-sigma line. fasttransforms.mqh needed its own include - verified that none of ap/optimization/statistics/solvers/linalg reaches it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 07:44:36 -04:00
for(int mm = 0; mm < ENS_MAX_MEMBERS; mm++)
g_ensVoteMember[row * ENS_MAX_MEMBERS + mm] = 0.0;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
g_ensVoteMask[row] = 0;
fix(gate): the ensemble gate certified a vote the EA never casts g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on the same criterion the live trade does". It did not. Two independent mismatches, both silent: CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100 number straight off the softmax head. Live contributes m_weight x the tier's pattern weight, and BOTH of those are rewritten from the signal DB by UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share an axis and nothing relates them, so the same bar was one number to the gate and a different one to the order path. Same shape as the 2026-08-09 geometry incident: certified on one game, paid on another. DENOMINATOR. The gate divided by the member count, so an abstaining member pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero contribution in BOTH the sum and the count (`if(direction == 0) continue;` before `number++`) - live is a mean over VOTERS. The gate was therefore scoring a strictly more agreement-heavy set of bars than the EA trades. The contribution hook's own comment asserted the opposite ("abstentions dilute the average exactly as they do in the live vote"), while the AI_CHOICE enum 20 lines away correctly documented union semantics. LiveVoteContribution() is now the single definition of "what this member votes", called from the gate; the live path reaches the same arithmetic through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually voted, separately from who evaluated the bar, because those are the divisor and the shared-population test respectively. ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar wrapper - the OOS scan holds the scanned bar's decision in a local, and dPrevSignal is a different bar. NOT changed, deliberately: the gate still does not model live NMS declustering, and the per-member solo gate still scores every directional call rather than threshold-clearing ones. Both are selection-metric changes and this codebase has twice been bitten by switching one blind. Also corrects a stale paragraph in m_pattern_0's declaration block quoting 80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has been since the confidence floor and alternation gate were removed; the block carried both tables at once, and the dead one was quoted back as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
g_ensVoteVoterMask[row] = 0;
feat(vote): thresholds become confidence percentages, on ONE scale everywhere User request: "the entry/exit thresholds are manual numbers, I would like them to be confidence percentages, so the current 20 would be only 20% confidence in a profitable trade." WHY 20 WAS EVER SENSIBLE. Under UseDatabaseRanking both factors of a filter's contribution are win rates: the pattern weight is that pattern's measured win rate (UpdateSignalsWeights -> ApplyPatternWeight) and m_weight is the filter's average win rate over its patterns, /100. Dividing the sum by the VOTER COUNT therefore produced a mean of PRODUCTS of two win rates - a genuinely 60%-accurate filter firing a 60% pattern scored 0.60 x 60 = 36. The number was never on a probability scale, so its magnitude meant nothing on its own. Dividing by Sum(m_weight) instead makes it a weighted MEAN of win rates, which is a win rate: result = Sum(w_i*p_i)/Sum(w_i). Every voter at 60% now reads 60; MACD's double-divergence pattern (weight 100) voting alone reads 100. m_weight stops being a discount on the probability and becomes how much a filter's opinion COUNTS - which is what a module weight should always have been. Default Min_Vote_Open 20 -> 50: not a tightening, the same bar re-expressed. ONE SCALE, EVERYWHERE - the part that made this bigger than a rescale. Three other places compared against a 0..1 softmax confidence and would each have become a fresh currency mismatch the moment the input changed meaning: * the AI early-exit route (LiveSignedConfidence vs m_ai_exit_threshold) now reads m_lastAiVote - the AI filters' own weighted mean, undiluted by the classic side, which is the only reason that route exists - against the same m_threshold_close the averaged vote uses. m_ai_exit_threshold is retired rather than left dangling. * m_oosDecisionSeries now carries the vote, not the confidence, so the exit SIMULATION stops modelling a close rule the EA does not run. * ExitPolicy() clamped anything > 1.0 to zero. Passing the unscaled input through that would have silently switched vote exits off in the simulation while live went on running them - found before it shipped; the bound now tracks the scale. LiveSignedConfidence() is deliberately untouched and still 0..1: MM sizing, SL/TP scaling and the intelligent trailing want a model confidence, not a win rate. CALIBRATION CAVEAT, stated in the code where the claim is made: this is only a real probability to the extent the pattern weights are. A pattern with fewer than MIN_TRADES_FOR_WIN_RATE journaled trades keeps its DEFAULT weight - a designed prior (25/50/75/100 for the AI tiers), not a measurement. Until the signal DB fills, "60" means "the designed conviction of the patterns that fired". Closing that gap is the next commit. Also corrects VOTE_CLOSE_PRESETS' comment, which documented the two scales this removes. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:52:08 -04:00
g_ensVoteWeightSum[row] = 0.0;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
//--- outcomes and label come from the shared label cache, so they are identical across
//--- members - whichever member reaches the bar first writes them
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
g_ensVoteWinLong[row] = winLong;
g_ensVoteWinShort[row] = winShort;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
g_ensVoteDirLabel[row] = dirLabel;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
}
if((g_ensVoteMask[row] & bit) != 0)
return; // already contributed to this bar this era (defensive - a re-run must not double-count)
fix(gate): the ensemble gate certified a vote the EA never casts g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on the same criterion the live trade does". It did not. Two independent mismatches, both silent: CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100 number straight off the softmax head. Live contributes m_weight x the tier's pattern weight, and BOTH of those are rewritten from the signal DB by UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share an axis and nothing relates them, so the same bar was one number to the gate and a different one to the order path. Same shape as the 2026-08-09 geometry incident: certified on one game, paid on another. DENOMINATOR. The gate divided by the member count, so an abstaining member pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero contribution in BOTH the sum and the count (`if(direction == 0) continue;` before `number++`) - live is a mean over VOTERS. The gate was therefore scoring a strictly more agreement-heavy set of bars than the EA trades. The contribution hook's own comment asserted the opposite ("abstentions dilute the average exactly as they do in the live vote"), while the AI_CHOICE enum 20 lines away correctly documented union semantics. LiveVoteContribution() is now the single definition of "what this member votes", called from the gate; the live path reaches the same arithmetic through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually voted, separately from who evaluated the bar, because those are the divisor and the shared-population test respectively. ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar wrapper - the OOS scan holds the scanned bar's decision in a local, and dPrevSignal is a different bar. NOT changed, deliberately: the gate still does not model live NMS declustering, and the per-member solo gate still scores every directional call rather than threshold-clearing ones. Both are selection-metric changes and this codebase has twice been bitten by switching one blind. Also corrects a stale paragraph in m_pattern_0's declaration block quoting 80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has been since the confidence floor and alternation gate were removed; the block carried both tables at once, and the dead one was quoted back as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
g_ensVoteSum[row] += signedVote;
feat(baselines): combining-weight fit, MLP cross-validation, all-lags correlation Three ALGLIB additions, all measurement-only and all under the existing Run_Alglib_Baselines switch. MinBLEIC COMBINING WEIGHTS. The live ensemble weights each member by its own pooled holdout win rate - a defensible prior, but not a fit, and nothing has ever asked what mixture minimises error on the bars the members disagreed about. Two individually-mediocre members wrong in different places can beat one individually better, and a per-member win rate cannot express that because it never looks at them jointly. Solved on the simplex (w >= 0, sum w = 1), which is exactly what MinBLEIC is for. Non-negative because a negative weight asserts "trade the opposite of this member", a claim ~60 effective observations cannot support. Least squares on the signed outcome rather than precision: precision is a STEP function of the threshold that no gradient method can walk, and optimising a smooth proxy for a step decision is how c3daded put every operating point 14pp underwater - so the result is reported in BOTH currencies, the SSE it minimised and the directional hit rate the mixture would actually have scored against the equal mix. If the second does not improve, the first is noise. This needed data that did not exist: g_ensVoteSum accumulates member contributions and the sum destroys the decomposition, while g_ensVoteVoterMask records only WHETHER a member voted, never what. g_ensVoteMember[] keeps them unsummed. The live arithmetic is untouched. ENS_MAX_MEMBERS is 8 and deliberately larger than MAX_AI_SIGNALS (5): independent caps, over-allocating is free, and matching them would make this array silently short the day the registry grows - a cap that has already dropped a member once without saying so. MLPKFoldCVLBFGS. Every baseline row carries a binomial SE, which is the sampling error of SCORING a fixed model and says nothing about how much the FIT moves. One LBFGS run from one random start can land anywhere, and a baseline that cleared or missed the bar on luck of initialisation reads exactly like one that did it on merit. 3 folds, because each is a full retrain. LBFGS not LM - LM builds a Hessian over ~7,700 weights. CCorr ALL-LAGS PROFILE. Added BESIDE the MI lag profile, not instead: MI catches nonlinear dependence and is the stronger negative, which is why it settled the verdict - but its per-lag permutation null limits it to ~20 lags. FFT correlation gets every lag in one O(n log n) pass, so linear structure parked at lag 300 would surface for free. Different question, not a replacement. Walks CONTIGUOUS bars, unlike everything else in this file, because a lag index is meaningless otherwise; both series are mean-centred first since CorrR1D is a raw sum of products; and the max over columns x lags is judged against a Sidak family of exactly that size, not a bare 2-sigma line. fasttransforms.mqh needed its own include - verified that none of ap/optimization/statistics/solvers/linalg reaches it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 07:44:36 -04:00
if(m_ensembleIndex >= 0 && m_ensembleIndex < ENS_MAX_MEMBERS)
g_ensVoteMember[row * ENS_MAX_MEMBERS + m_ensembleIndex] = signedVote;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
g_ensVoteMask[row] |= bit;
//--- VOTER, not merely present. -0.0 compares equal to 0.0, so an abstention that arrived
//--- with a negative zero is still correctly excluded here.
feat(vote): CONSENSUS arithmetic - agreement is now what the threshold dials Era-680 report, all three observations one equation: "peak 29, no arrows at threshold 30" / "at 20, arrows on EVERY bar" / "label at 12 while arrows everywhere". Under the voters-only divisor, any bar with at least one directional voter read the weighted mean of the firing tiers' weights - and once the tiers self-ranked to each model's pooled win rate (~28-31), that mean was NEAR-CONSTANT regardless of headcount. One member alone: ~29. Four unanimous: ~29. Min_Vote_Open was a step function around that constant - above it nothing ever fired, below it everything did - and the label's 12 was a 3v1 split netting through the same divisor. Not three display bugs: one arithmetic that could not express agreement. The divisor is now the CAPABLE weight - every filter that could vote, whether it did or not: * live (Direction): VoteCapableWeight() - classic pattern ladders always, veto filters never, AI members once past the same readiness test LongCondition gates on. A model still training must not dilute an ensemble it cannot join: four trainees + one deployed model is a solo chart wearing an ensemble label, and the solo vote reads full strength. * gate (EnsembleEraVerdict): g_ensVoteWeightSum accumulates for every member that EVALUATED the bar, Neutral included. * overlay sweep + prospective readout: weight counts whenever the member has data; a snapshotted Neutral dilutes. One arithmetic, four sites, same numbers everywhere. What the numbers become (four members, w~0.29, tiers~29): unanimous ~29 - the CEILING, which is the pooled win rate and is what the peak displays; 3-of-4 ~22; 2-of-4 ~14.5; 3v1 ~14.5. Min_Vote_Open 20 now means "roughly three-quarters of the ensemble's trust agrees, net". It MUST sit below the ceiling to ever fire - the census/peak states the ceiling. This is the ensemble the user specified in the original design discussion ("if the perceptron also votes, both together reach the threshold; if another NN votes the other side, the threshold is not reached") - union semantics was the pre-ensemble behaviour, kept until measurement showed its vote magnitude was a constant. Plus overlay DECLUSTERING, the other half of "arrows on every bar": the same three NMS rules as the per-member arrows (same-direction runs collapse to their first bar, cross-direction flicker keeps the stronger side), online over the sweep's strictly oldest->newest walk. Suppression is a verdict and deletes a standing arrow; the den==0 no-data skip still never does. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 06:58:03 -04:00
g_ensVoteWeightSum[row] += voteWeight;
fix(gate): the ensemble gate certified a vote the EA never casts g_ensembleVoteThreshold's comment claims the combined-vote scorer "fires on the same criterion the live trade does". It did not. Two independent mismatches, both silent: CURRENCY. Each member contributed its raw signed confidence x100 - a 33..100 number straight off the softmax head. Live contributes m_weight x the tier's pattern weight, and BOTH of those are rewritten from the signal DB by UpdateSignalsWeights(). A head output and a DB-ranked win-rate weight share an axis and nothing relates them, so the same bar was one number to the gate and a different one to the order path. Same shape as the 2026-08-09 geometry incident: certified on one game, paid on another. DENOMINATOR. The gate divided by the member count, so an abstaining member pulled the average toward zero. CExpertSignalCustom::Direction() skips a zero contribution in BOTH the sum and the count (`if(direction == 0) continue;` before `number++`) - live is a mean over VOTERS. The gate was therefore scoring a strictly more agreement-heavy set of bars than the EA trades. The contribution hook's own comment asserted the opposite ("abstentions dilute the average exactly as they do in the live vote"), while the AI_CHOICE enum 20 lines away correctly documented union semantics. LiveVoteContribution() is now the single definition of "what this member votes", called from the gate; the live path reaches the same arithmetic through LongCondition/ShortCondition. g_ensVoteVoterMask records who actually voted, separately from who evaluated the bar, because those are the divisor and the shared-population test respectively. ConfidenceTier() is split into ConfidenceTierFor(signal) plus a thin live-bar wrapper - the OOS scan holds the scanned bar's decision in a local, and dPrevSignal is a different bar. NOT changed, deliberately: the gate still does not model live NMS declustering, and the per-member solo gate still scores every directional call rather than threshold-clearing ones. Both are selection-metric changes and this codebase has twice been bitten by switching one blind. Also corrects a stale paragraph in m_pattern_0's declaration block quoting 80/87/93/100 as the tier defaults. The constructor is 25/50/75/100 and has been since the confidence floor and alternation gate were removed; the block carried both tables at once, and the dead one was quoted back as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:23:33 -04:00
if(signedVote != 0.0)
g_ensVoteVoterMask[row] |= bit;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
g_ensVoteCursor[m_ensembleIndex] = row + 1;
}
//--- This member's just-finished era, held until the ensemble verdict can act on it. Same
//--- quantities the solo gate keeps in m_best*.
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
double m_eraStatPrecPct;
double m_eraStatChancePct;
int m_eraStatCalls;
bool m_eraStatTradeable;
bool m_eraStatTwoSided;
double m_eraStatScore;
double m_eraStatBlended;
double m_eraStatThreshold;
//--- Which era this member's in-memory snapshot belongs to (-1 = none).
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
long m_checkpointEra;
void EnsembleStashEraStats(const double precPct, const double chancePct, const int calls,
const bool tradeable, const bool twoSided, const double score,
const double blended)
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
{
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
m_eraStatPrecPct = precPct;
m_eraStatChancePct = chancePct;
m_eraStatCalls = calls;
m_eraStatTradeable = tradeable;
m_eraStatTwoSided = twoSided;
m_eraStatScore = score;
m_eraStatBlended = blended;
//--- the operating point belongs with the weights it was fitted for - see m_bestDirConfThreshold
m_eraStatThreshold = m_dirConfThreshold;
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
}
//--- Called once per era from the era-end block, after this member's statistics are final.
refactor(dry): one binomial arithmetic for every "is this edge real" test The formula p(1-p)/n was transcribed nine times across six files - the two deploy gates, the two edge floors, the collapse recall floor, the barrier rung ladder, the inference bin SE, the pooled inverse-variance weights and both detectability reports. System\BinomialStats.mqh now holds it once, as free functions with no class dependency, so the god-class declaration does not grow to host pure math. BinomialVar(p, n) p(1-p)/n BinomialSEPct(p, n) 100*sqrt(p(1-p)/n) BinomialCallsForEdge(p, edge, sigmas) the same, solved for n NormalUpperTailQ(z) Q(z), via Math\Stat\Normal.mqh SidakFamilyP(z, N) 1-(1-Q(z))^N Value-preserving by construction: rates go in as probabilities so no call site gained a *100/100 round-trip, and BinomialSEPct is written through BinomialVar so the multiply order is the one it replaced. Every degenerate guard each site carried (p<=0, p>=1, n<=0) now lives in one place and returns the 0 those sites already treated as "no bar to clear". CExpertSignalAIBase::NormalUpperTail is gone; NormalUpperTailQ replaces it. What consolidating SURFACED, and is deliberately NOT changed here: the two Sidak selection gates compute their SE on the RAW call count, while every other SE in the project deflates by EffectiveSampleSize() for triple- barrier label overlap. That makes them the most permissive test in the codebase, by ~sqrt(mean label lifespan). Correcting it tightens a live deploy bar, which is a policy decision, not a refactor - flagged in the code at both sites. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 19:50:24 -04:00
//--- Defined in Training.mqh - it needs the PLATEAU_* machinery.
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
void EnsembleOosPassComplete(const long votedEra, double &etaLocal);
//--- The verdict itself, and its pieces. needMask names the members whose reads the vote is built
//--- from (still-training members only - a paused or deployed member is not voting in training).
void EnsembleEraVerdict(const int needMask, const long votedEra, double &etaLocal);
void EnsembleCommitJointCheckpoint(const long votedEra);
//--- Does the best combined-vote era survive having been CHOSEN out of g_ensCandidateEras eras?
//--- Identical construction to BestCheckpointSurvivesSelection, applied to the vote.
bool EnsembleSurvivesSelection(double &zObs, double &pFamily, int &nTried);
//--- Single choke point for this signal's on-chart status text. Every AI-side SetStatusLabel
//--- call site routes through here so no mode can regress into four stacked panels.
void PublishStatus(const string text, const bool force = false)
{
if(!m_ensembleMember)
{
SetStatusLabel(text);
return;
}
//--- Called EVERY publish, not just the first. The old "claim once, first publisher wins the
//--- next free row" form is what ordered the panel by who was busiest instead of by member
//--- index.
fix(indicators+panel): the dead handle is MEASURED now - recreate it; and order the ensemble panel by member, not by who published first THE ANSWER, off the instrumentation added in be39674, first run: ConvLSTM [HYB-2484]: TUNABLE INDICATOR REPORTS NO CALCULATED BARS - 1 tunable indicator(s) enabled and the least-ready answers BarsCalculated()=-1 ... Per-indicator depth: price=33982 MA=-1 ZigZag=33982 ATR=33982 MA=-1 with price, ZigZag and ATR all at full depth. **The handle is INVALID, not short.** Same line on USDJPY (price=50179 MA=-1). Depth was never the problem; the previous session's five theories were all answering the wrong question. And it is per-member, not per-chart: LSTM-2484 ran the 34-candidate auto-tune on that same chart at 15:24:18 and went on to train normally (feature health, 51 features, excursion head) reading the same indicator. Only ConvLSTM's handle - the last member constructed - was dead. WHY is still not established. All four members request ADMovingAverage with identical params, so MT5 hands them the SAME refcounted handle, and the tuner's inner loop is Create-then-IndicatorRelease over exactly that shared handle; that is the obvious suspect and it is NOT yet proven, so this commit does not act on it. 1. IndicatorDepthReport() NOW PRINTS HANDLE NUMBERS, not just depths. "MA=-1" says the handle is dead. "MA=-1(h12)" against another member's "MA=33982 (h12)" says it is the SAME handle and someone released it; "(h-1)" says it was never created. That is the difference between a refcount bug and a creation failure and it is one field. This is the measurement the shared-handle suspicion needs before anyone acts on it. 2. RECREATE A DEAD HANDLE INSTEAD OF SWEEPING AGAINST IT. A member that cannot read its own indicator must rebuild it. RepairDeadIndicatorHandles() re-Creates only the ENABLED tunables reporting BarsCalculated() < 0 - a merely COLD indicator (valid handle, 0 bars) is left alone to warm up the normal way. It does NOT release first: -1 means the terminal no longer knows the handle, so there is nothing to give back, and MT5 recycles handle VALUES so releasing a stale one could decrement whatever now owns that number. 30s cooldown, because every ServableBars() consumer reaches it including live inference on every tick. The feature cache is dropped with it, and the log names before/after depths. Cause-agnostic on purpose. Whatever is killing the handle, sweeping 50,163 bars against a buffer that answers EMPTY_VALUE at every index - then discarding the era and doing it again - is not a recovery. 3. THE SWEEP NOW HOLDS ON A DEAD HANDLE. ServableBars() keeps answering `want` (its contract; live inference and online learning have their own refusal paths and a 0 there reads as "no history at all"). SettledBars() - the training sweep's entry, the one caller that can afford to wait - returns 0 instead, so Train() holds and reports rather than burning a full-history pass it is guaranteed to throw away. A recreated handle is cold, so it primes through the existing settle path on the next call. If the repair fails the member holds indefinitely and says so every minute, and be39674's barrier liveness escape releases the rest of the ensemble after 12 minutes - which is the correct degradation and is exactly what the log shows happening. 4. THE PANEL ROWS WERE ORDERED BY WHO PUBLISHED FIRST. Reported on XAUUSD: LSTM, ConvLSTM, Perceptron, Convolutional instead of Perceptron, Convolutional, LSTM, ConvLSTM. ClaimEnsemblePanelSlot() handed out the next free row on each member's FIRST PublishStatus() call, so the order was a race - the members busy sweeping published before the ones sitting idle at the era barrier, and be39674 sharpened it by (correctly) making a held member stop writing the terse line. Rows are now keyed to m_ensembleIndex, the registration/construction order, which is fixed for the life of the chart. Claimed on every publish rather than once, so it is idempotent and refreshes the tag for a member whose ID was not final when it first published (the config-tag suffix is appended during InitIndicators, after EnsembleMember() registers). Unclaimed rows are skipped by the render and excluded from the model count, so a member that has not published yet leaves no gap and shifts nobody. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 15:40:00 -04:00
m_ensemblePanelSlot = ClaimEnsemblePanelSlot(DisplayName(), m_ensembleIndex);
int nl = StringFind(text, "\n");
PublishEnsembleStatus(m_ensemblePanelSlot, (nl > 0) ? StringSubstr(text, 0, nl) : text, force);
}
void EnableOnlineLearning(bool value) { m_enableOnlineLearning = value; }
//--- Exit policy, pushed in from Warrior_EA.mq5 so the gate grades the same rule the live path
//--- runs. voteThreshold is Signal_ThresholdClose UNSCALED, on the same 0-100 confidence scale
//--- as the live close threshold (>100 disables it by arithmetic, exactly as live does).
fix(shutdown): make ExitPolicy public, and stop every long loop the moment MT5 asks Two things, one of which was a compile error. 1. ExitPolicy() was declared in the protected block but is pushed in from Warrior_EA.mq5:770. Moved to public beside the other EA-facing setters. 2. Chart objects surviving OnDeinit. The 4,500 ms teardown budget is measured from the STOP REQUEST, not from OnDeinit's first line, and OnDeinit cannot begin until whatever is in flight returns - so a scan still running after _StopFlag is raised does not delay the cleanup, it SPENDS it, and the purge never gets its turn. New CExpertSignalAIBase::ShutdownRequested() = IsStopped() || m_shutdownInProgress. Deliberately NOT m_trainingStopRequested: that latches, and a latched flag would permanently disable scans that must run again on the next Start. Guarded, longest first: - TuneIndicatorsByFilter - per candidate, restoring the OPERATOR's settings on the way out (best[] is mutated in place; the tuner otherwise keeps the last trial's parameters, which nothing chose). - ReportBarrierGeometryScan - per pairing, breaking to ONE restore point so m_barrierScanLiveLabels can never be left true (that makes ComputeLabelForBar read the last candidate's multiples as the configured geometry). - ReportFeatureLabelInformation / ReportExcursionInformation / lag profile - nulls ABANDON rather than truncate: fewer draws is not a smaller null, it is a wrong one, and p shifts toward significance. m_dirEvidence staying false is the safe direction. - SimulateExitPolicyOutcomes - zeroes its accumulators so the divergence line is dropped instead of latching a partial expectancy as the run's only report. - ReportGeometryExpectancyScan - per ladder rung. - HttpGet - one choke point for up to a dozen blocking WebRequests per first-pass Update(). An in-flight request cannot be cancelled; refusing to start another is the whole remedy. - PollTraining, OnChartEventHandler's study event, TuneIndicatorsAndTrain - entry points, so a queued event cannot open an era during teardown. TuneIndicatorsAndTrain's guard is the first statement, ahead of the m_tuneFilterDone / g_ensembleChartTuneDone latches. - OnTick / OnTimer / OnChartEvent. Training's own bar loops already honoured this (pass 1 per bar, passes 2/2.5/3 yield on a 120 ms budget); the warm-up scans did not, and they are the longest uninterruptible stretches the EA has. StopTraining() is unchanged: the operator's Stop still finalises synchronously. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 17:03:11 -04:00
void ExitPolicy(double voteThreshold, bool holdToBarrier)
{
feat(vote): thresholds become confidence percentages, on ONE scale everywhere User request: "the entry/exit thresholds are manual numbers, I would like them to be confidence percentages, so the current 20 would be only 20% confidence in a profitable trade." WHY 20 WAS EVER SENSIBLE. Under UseDatabaseRanking both factors of a filter's contribution are win rates: the pattern weight is that pattern's measured win rate (UpdateSignalsWeights -> ApplyPatternWeight) and m_weight is the filter's average win rate over its patterns, /100. Dividing the sum by the VOTER COUNT therefore produced a mean of PRODUCTS of two win rates - a genuinely 60%-accurate filter firing a 60% pattern scored 0.60 x 60 = 36. The number was never on a probability scale, so its magnitude meant nothing on its own. Dividing by Sum(m_weight) instead makes it a weighted MEAN of win rates, which is a win rate: result = Sum(w_i*p_i)/Sum(w_i). Every voter at 60% now reads 60; MACD's double-divergence pattern (weight 100) voting alone reads 100. m_weight stops being a discount on the probability and becomes how much a filter's opinion COUNTS - which is what a module weight should always have been. Default Min_Vote_Open 20 -> 50: not a tightening, the same bar re-expressed. ONE SCALE, EVERYWHERE - the part that made this bigger than a rescale. Three other places compared against a 0..1 softmax confidence and would each have become a fresh currency mismatch the moment the input changed meaning: * the AI early-exit route (LiveSignedConfidence vs m_ai_exit_threshold) now reads m_lastAiVote - the AI filters' own weighted mean, undiluted by the classic side, which is the only reason that route exists - against the same m_threshold_close the averaged vote uses. m_ai_exit_threshold is retired rather than left dangling. * m_oosDecisionSeries now carries the vote, not the confidence, so the exit SIMULATION stops modelling a close rule the EA does not run. * ExitPolicy() clamped anything > 1.0 to zero. Passing the unscaled input through that would have silently switched vote exits off in the simulation while live went on running them - found before it shipped; the bound now tracks the scale. LiveSignedConfidence() is deliberately untouched and still 0..1: MM sizing, SL/TP scaling and the intelligent trailing want a model confidence, not a win rate. CALIBRATION CAVEAT, stated in the code where the claim is made: this is only a real probability to the extent the pattern weights are. A pattern with fewer than MIN_TRADES_FOR_WIN_RATE journaled trades keeps its DEFAULT weight - a designed prior (25/50/75/100 for the AI tiers), not a measurement. Until the signal DB fills, "60" means "the designed conviction of the patterns that fired". Closing that gap is the next commit. Also corrects VOTE_CLOSE_PRESETS' comment, which documented the two scales this removes. NOT COMPILED - user compiles in MetaEditor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 15:52:08 -04:00
m_exitVoteThreshold = (voteThreshold > 100.0) ? 0.0 : voteThreshold;
fix(shutdown): make ExitPolicy public, and stop every long loop the moment MT5 asks Two things, one of which was a compile error. 1. ExitPolicy() was declared in the protected block but is pushed in from Warrior_EA.mq5:770. Moved to public beside the other EA-facing setters. 2. Chart objects surviving OnDeinit. The 4,500 ms teardown budget is measured from the STOP REQUEST, not from OnDeinit's first line, and OnDeinit cannot begin until whatever is in flight returns - so a scan still running after _StopFlag is raised does not delay the cleanup, it SPENDS it, and the purge never gets its turn. New CExpertSignalAIBase::ShutdownRequested() = IsStopped() || m_shutdownInProgress. Deliberately NOT m_trainingStopRequested: that latches, and a latched flag would permanently disable scans that must run again on the next Start. Guarded, longest first: - TuneIndicatorsByFilter - per candidate, restoring the OPERATOR's settings on the way out (best[] is mutated in place; the tuner otherwise keeps the last trial's parameters, which nothing chose). - ReportBarrierGeometryScan - per pairing, breaking to ONE restore point so m_barrierScanLiveLabels can never be left true (that makes ComputeLabelForBar read the last candidate's multiples as the configured geometry). - ReportFeatureLabelInformation / ReportExcursionInformation / lag profile - nulls ABANDON rather than truncate: fewer draws is not a smaller null, it is a wrong one, and p shifts toward significance. m_dirEvidence staying false is the safe direction. - SimulateExitPolicyOutcomes - zeroes its accumulators so the divergence line is dropped instead of latching a partial expectancy as the run's only report. - ReportGeometryExpectancyScan - per ladder rung. - HttpGet - one choke point for up to a dozen blocking WebRequests per first-pass Update(). An in-flight request cannot be cancelled; refusing to start another is the whole remedy. - PollTraining, OnChartEventHandler's study event, TuneIndicatorsAndTrain - entry points, so a queued event cannot open an era during teardown. TuneIndicatorsAndTrain's guard is the first statement, ahead of the m_tuneFilterDone / g_ensembleChartTuneDone latches. - OnTick / OnTimer / OnChartEvent. Training's own bar loops already honoured this (pass 1 per bar, passes 2/2.5/3 yield on a 120 ms budget); the warm-up scans did not, and they are the longest uninterruptible stretches the EA has. StopTraining() is unchanged: the operator's Stop still finalises synchronously. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 17:03:11 -04:00
m_exitHoldToBarrier = holdToBarrier;
}
void MaxErasPerRun(int value) { m_maxErasPerRun = value; }
void OOSSplit(int value) { m_oosSplitPct = value; }
2026-08-19 18:55:36 -04:00
//--- No setters for m_historyBars / m_minTrainYear. The window is DERIVED at InitNeuralNetwork or
//--- ADOPTED from the .cfg (see DeriveHistoryBars); the year floor is a constructor constant. Both
//--- remain members only because the .cfg field layout is positional.
void UseVolumes(bool value) { m_useVolumes = value; }
void UseTime(bool value) { m_useTime = value; }
void UseATR(bool value) { m_useATR = value; }
void UseMA(bool value) { m_useMA = value; }
void UseRSI(bool value) { m_useRSI = value; }
void UseMACD(bool value) { m_useMACD = value; }
void UseIchimoku(bool value) { m_useIchimoku = value; }
void UseSwingContext(bool value) { m_useSwingContext = value; }
void UseNews(bool value) { m_useNews = value; }
void NewsFeatureWindowMinutes(int value) { m_newsFeatureWindowMinutes = value; }
fix(signals): revive a dead MA model, and demote Sanyaku from state to event Two defects surfaced by research/test_classic.py, both verified fixed by re-running the transcription against 178k bars of EURUSD H1. CSignalMA model 1 could never fire. For any recursive average - and MA_TYPE_EMA is the shipped default - MA(i) = a*Close(i) + (1-a)*MA(i+1), so DiffMA(i) = a * (Close(i) - MA(i+1)) DiffCloseMA(i) = (1-a) * (Close(i) - MA(i+1)) are positive multiples of one quantity and always share a sign. Model 1 asks for a close BELOW a RISING average, which is precisely the combination that identity forbids: 0.000% of bars, either direction, any symbol. The MQL5 standard library this was ported from defaults to MODE_SMA, where the two are merely correlated - the bug arrived with the EMA default, not with the port. Reading the slope one bar back (DiffMAPrev) breaks the tie for every MA type while keeping the model's stated meaning. Now fires on 7.92% of bars. CSignalIchimoku model 11 fired on 27% of bars at weight 100. Sanyaku is three standing STATES conjoined with no transition term, so it held across long stretches - and being last in the if-chain at the top weight, the module's highest-conviction reading was also its most common one, overwriting all eight event models below it on a quarter of all bars. The old comment rejected an event form because "demanding all three flip on the same bar would fire almost never" - true, but that is not the alternative. Kouten is the TURN: the ALIGNMENT transitions, and only one role need change for it to. Testing !Sanyaku(idx+1) fires once per aligned stretch. Now 2.17%, in line with Kumo breakout (2.4%) and the strong TK cross (1.1%). DataReady() extended one bar deeper to cover the lookback. Neither pattern showed edge before or after; this is about the models meaning what they say and the vote not being dominated by a constant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:14:34 -04:00
void UseCrossAsset(bool value) { m_useCrossAsset = value; }
feat(ai): spread as a volatility-regime feature, and fix a stale-index cache in both new blocks Adds spread/ATR and the spread change ratio as network inputs (EnableSpreadFeature, default on). Spread is the one microstructure channel that is both FX-available and genuinely historical in the Strategy Tester - "during testing, the spread is not modeled but is taken from historical data" - so unlike swap, signed tick flow or depth of market it is something a backtest can honestly validate. What it encodes, stated precisely because the raw measurement overstates it. research/test_spread.py found spr/atr the strongest single feature in this codebase, on 5 of 8 instrument/geometry cells at 2-4x any volume feature. But the barrier LABEL charges the spread inside its own barriers, so a wide-spread bar is mechanically likelier to resolve as a loss and the feature would partly be predicting its own cost model. Relabelling at zero cost and re-measuring the identical feature showed 20-40% of it WAS that tautology and the majority was not (XAUUSD retained 97%). What survives is a volatility-regime reading: spread is near-fixed while ATR is not, so the ratio runs high exactly when realised volatility is below its own ATR estimate, which genuinely predicts whether ATR-scaled barriers get reached. It is UNSIGNED - Neutral-vs-directional only, never a side. Also fixes a stale-index bug I introduced with the cross-asset panel and had just repeated in the spread series. Both cached on length alone: if(m_crossAsset.Bars() >= bars) return true; MQL5 series indices are relative to NOW, so one new closed candle shifts every index by one. Keyed only on length, the panel keeps serving its index 0 as a bar that is no longer the newest, and every cross-asset value is read one bar out of step with the price features sitting beside it in the same vector - silently, with no error and no shape change. This is the same class of defect as the dtStudied watermark behind the zero-direction backtests. Both now carry a datetime anchor on m_Time.GetData(0), the same invalidation key the label/feature bar caches already use. And a performance fix that fell out of it: with correct invalidation the panel rebuilds on every new bar, and RefreshConvergedSignal runs per bar - which in the tester would mean one full multi-symbol resample per simulated bar at training depth. Inference only reads bars 0..m_historyBars-1 plus the panel's own slow window, so it now requests exactly that. The cache check is >=, so a deeper panel left from training still satisfies it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:42:40 -04:00
void UseSpreadFeature(bool value) { m_useSpreadFeature = value; }
void UseADCumulativeDelta(bool value) { m_useADCumulativeDelta = value; }
void UseADShorteningOfThrust(bool value) { m_useADShorteningOfThrust = value; }
void UseADWyckoffEventStream(bool value) { m_useADWyckoffEventStream = value; }
void UseADWyckoffFailedStructure(bool value) { m_useADWyckoffFailedStructure = value; }
void UseADWyckoffSignificantBarInversion(bool value) { m_useADWyckoffSignificantBarInversion = value; }
void AutoTuneIndicators(bool value) { m_autoTuneIndicators = value; }
void UseAltData(bool value) { m_altDataEnabled = value; }
//--- control-panel API (Warrior_EA.mq5): current-config-only training/weights control.
//--- "current config" == this signal instance's own m_fileName (symbol+period+id+topology),
//--- never touches another signal type's or another symbol/timeframe's saved files.
void PauseTraining(void) { m_trainingPaused = true; PrintVerbose(ID + ": training paused by user (era " + IntegerToString(m_eraCount) + ")"); }
void ResumeTraining(void) { m_trainingPaused = false; PrintVerbose(ID + ": training resumed by user (era " + IntegerToString(m_eraCount) + ")"); }
bool IsTrainingPaused(void) const { return m_trainingPaused; }
bool IsTrainingStopped(void) const { return m_trainingStopRequested; }
bool TrainingComplete(void) const { return m_trainingComplete; }
fix(deinit): a full model write was running ahead of the cheap cleanup "Abnormal termination" is back, and this time it is not the arrows. The timing names the culprit exactly: 16:02:31.547 OnDeinit: shutting down 16:02:36.003 Abnormal termination <- 4.46 s, MetaTrader gave up 16:02:36.226 chart signals - persisted <- cleanup finished 0.2 s LATE OnDeinit called StopTraining() BEFORE the chart cleanup. StopTraining() finalises an in-flight run, and FinalizeTrainRun() restores the best checkpoint and then persists it - a full ~1MB model write per signal. So the expensive step ran ahead of the cheap bounded one, which is precisely the inversion the shutdown ordering exists to prevent. The previous fix put PersistWeightsOnShutdown last and missed that StopTraining smuggles a second save in at the front. Two changes: Cleanup now runs FIRST, then StopTraining, then the weight save. The visible teardown is cheap and bounded, so it always completes even when everything after it is killed. And the deploy-persist inside FinalizeTrainRun is suppressed during shutdown. RestoreWeights() is an in-MEMORY swap, so the best checkpoint is already the live net by that line, and PersistWeightsOnShutdown writes exactly those weights moments later. The old path wrote the same model twice per signal - eight full writes across four charts - for no benefit. A user-pressed Stop still persists immediately, because nothing else would. Compiles 0 errors / 0 warnings. Build tag deinit-order-v2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 16:06:40 -04:00
//--- Set by OnDeinit before it calls StopTraining(), so FinalizeTrainRun() can tell a user-pressed Stop
//--- (persist the deployed model now - nothing else will) from a shutdown (PersistWeightsOnShutdown is
//--- moments away and writes the same bytes). See the guard in FinalizeTrainRun.
void MarkShutdown(void) { m_shutdownInProgress = true; }
//--- THE ONE QUESTION every long loop in this class must ask: has this program been asked to
//--- stop? These two are terminal - once either is true the program is going away.
fix(shutdown): make ExitPolicy public, and stop every long loop the moment MT5 asks Two things, one of which was a compile error. 1. ExitPolicy() was declared in the protected block but is pushed in from Warrior_EA.mq5:770. Moved to public beside the other EA-facing setters. 2. Chart objects surviving OnDeinit. The 4,500 ms teardown budget is measured from the STOP REQUEST, not from OnDeinit's first line, and OnDeinit cannot begin until whatever is in flight returns - so a scan still running after _StopFlag is raised does not delay the cleanup, it SPENDS it, and the purge never gets its turn. New CExpertSignalAIBase::ShutdownRequested() = IsStopped() || m_shutdownInProgress. Deliberately NOT m_trainingStopRequested: that latches, and a latched flag would permanently disable scans that must run again on the next Start. Guarded, longest first: - TuneIndicatorsByFilter - per candidate, restoring the OPERATOR's settings on the way out (best[] is mutated in place; the tuner otherwise keeps the last trial's parameters, which nothing chose). - ReportBarrierGeometryScan - per pairing, breaking to ONE restore point so m_barrierScanLiveLabels can never be left true (that makes ComputeLabelForBar read the last candidate's multiples as the configured geometry). - ReportFeatureLabelInformation / ReportExcursionInformation / lag profile - nulls ABANDON rather than truncate: fewer draws is not a smaller null, it is a wrong one, and p shifts toward significance. m_dirEvidence staying false is the safe direction. - SimulateExitPolicyOutcomes - zeroes its accumulators so the divergence line is dropped instead of latching a partial expectancy as the run's only report. - ReportGeometryExpectancyScan - per ladder rung. - HttpGet - one choke point for up to a dozen blocking WebRequests per first-pass Update(). An in-flight request cannot be cancelled; refusing to start another is the whole remedy. - PollTraining, OnChartEventHandler's study event, TuneIndicatorsAndTrain - entry points, so a queued event cannot open an era during teardown. TuneIndicatorsAndTrain's guard is the first statement, ahead of the m_tuneFilterDone / g_ensembleChartTuneDone latches. - OnTick / OnTimer / OnChartEvent. Training's own bar loops already honoured this (pass 1 per bar, passes 2/2.5/3 yield on a 120 ms budget); the warm-up scans did not, and they are the longest uninterruptible stretches the EA has. StopTraining() is unchanged: the operator's Stop still finalises synchronously. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 17:03:11 -04:00
bool ShutdownRequested(void) const { return (IsStopped() || m_shutdownInProgress); }
fix(reset): say what the reset actually did, per member and per file The user reports "Delete & Reset Weights only wipes the first NN". I could not find a code path that skips ensemble members, and I am not going to assert one: the handler loops g_aiSignals[0..g_aiSignalCount), all four topologies register unconditionally in OnInit, and SetIdentity gives each its own State\<id>\ folder so the six deleted paths are genuinely distinct per member. What IS true is that the whole success path was SILENT - six FileDelete calls per member printing only on failure, and one chart-wide Alert - so a four-member reset and a one-member reset produce byte-identical output. The symptom could be neither confirmed nor refuted from a log. That is the defect I can fix today. - COMPILED <timestamp> (__DATETIME__) beside the build tag. The hand-edited tag had sat at scan-nofwd-v5 across a week of commits, so it could not answer the question it exists for. The compile stamp cannot be forgotten. Tag bumped to reset-census-v6. - RegistryLine() (public): ID, active file path, common/local, era, deployed vs training, ensemble index. The reset handler prints a numbered census of the whole registry BEFORE the confirm dialog. If that says 1 on an AI_HYBRID chart the fault is registration, not the reset - and RegisterAISignal already has a loud MAX_AI_SIGNALS message for exactly that. - The confirmation dialog now names the count, so a wrong registry is visible before anything is deleted rather than after. - ResetWeights prints one line per member: N deleted / N already absent / N FAILED, plus a per-suffix breakdown. "absent" on a member that should have had a .nnw is a completely different fault from "deleted"; they were identical. - ResetWeights' return value was discarded. A member whose BuildFreshTopology fails has had its files deleted and has no network - and the Alert still said "weights reset". Counted now, with an INCOMPLETE alert when they disagree. - Same for dbm.ResetDatabase(), whose bool was also dropped. The DB is one shared file for every signal on the chart, so there is nothing per-member to loop - the log now says that explicitly, since it is the question being asked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 17:57:26 -04:00
//--- ONE LINE OF IDENTITY, for the census Warrior_EA.mq5 prints before it acts on g_aiSignals[].
//--- This makes them different.
fix(reset): say what the reset actually did, per member and per file The user reports "Delete & Reset Weights only wipes the first NN". I could not find a code path that skips ensemble members, and I am not going to assert one: the handler loops g_aiSignals[0..g_aiSignalCount), all four topologies register unconditionally in OnInit, and SetIdentity gives each its own State\<id>\ folder so the six deleted paths are genuinely distinct per member. What IS true is that the whole success path was SILENT - six FileDelete calls per member printing only on failure, and one chart-wide Alert - so a four-member reset and a one-member reset produce byte-identical output. The symptom could be neither confirmed nor refuted from a log. That is the defect I can fix today. - COMPILED <timestamp> (__DATETIME__) beside the build tag. The hand-edited tag had sat at scan-nofwd-v5 across a week of commits, so it could not answer the question it exists for. The compile stamp cannot be forgotten. Tag bumped to reset-census-v6. - RegistryLine() (public): ID, active file path, common/local, era, deployed vs training, ensemble index. The reset handler prints a numbered census of the whole registry BEFORE the confirm dialog. If that says 1 on an AI_HYBRID chart the fault is registration, not the reset - and RegisterAISignal already has a loud MAX_AI_SIGNALS message for exactly that. - The confirmation dialog now names the count, so a wrong registry is visible before anything is deleted rather than after. - ResetWeights prints one line per member: N deleted / N already absent / N FAILED, plus a per-suffix breakdown. "absent" on a member that should have had a .nnw is a completely different fault from "deleted"; they were identical. - ResetWeights' return value was discarded. A member whose BuildFreshTopology fails has had its files deleted and has no network - and the Alert still said "weights reset". Counted now, with an INCOMPLETE alert when they disagree. - Same for dbm.ResetDatabase(), whose bool was also dropped. The DB is one shared file for every signal on the chart, so there is nothing per-member to loop - the log now says that explicitly, since it is the question being asked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 17:57:26 -04:00
string RegistryLine(void) const
{
return StringFormat("%s | %s (%s) | era %d | %s%s", ID, m_activeFileName,
(m_activeFileCommon ? "common" : "local"), (int)m_eraCount,
(m_trainingComplete ? "deployed" : "training"),
(m_ensembleMember
? StringFormat(" | ensemble member %d", m_ensembleIndex) : " | solo"));
}
fix: flush the in-flight era on shutdown; sweep orphaned chart objects on attach Chart objects live in the MT5 chart PROFILE, not in this EA's files. They survive a terminal restart, a recompile, and deleting every .nnw/.cfg/.stats/.arrows on disk. Only a deinit that RUNS TO COMPLETION removes them - and MetaTrader force-terminates OnDeinit at roughly 4,500 ms, so a run killed mid-cleanup orphans them permanently with no owner left to clean up after. That is the "deleted every file, recompiled, restarted, old arrows and a stale panel still there" report: nothing was wrong with the files and deleting them could not have helped. Both halves are fixed. STOP OVERRUNNING THE BUDGET. OnDeinit used to finalise the in-flight run (StopTraining -> FinalizeTrainRun: checkpoint restore, live-state re-seed) and then write two full nets per chart. On four charts that is the bulk of the budget, spent to preserve a PARTIAL era that was never scored, never checkpointed and never deployable. FlushTrainRun() discards it instead - drop the resumable bookkeeping, leave the net neutral (unfreeze BN, flush the batch, batch size 1), skip the save - and training resumes from the last completed era, which the era-end save and the periodic autosave have already put on disk. What is discarded is bounded by one era. A CONVERGED model keeps the old finalise-and-save path: its weights can carry online-learning updates made since the last era boundary, and for a deployed model no further era boundary is coming to persist them. MAKE CLEANUP SELF-HEALING. Every purge sat behind a branch - no model loaded, sidecar missing - so the common paths returned leaving whatever the previous instance stranded. LoadChartSignals now sweeps the arrow namespace unconditionally before restoring, so the post-init chart holds exactly what the sidecar holds whichever branch runs, and the panel gets the same treatment before Create() (CAppDialog namespaces its controls, so a killed Destroy strands the lot and the next attach draws a second panel on the corpse). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:08:40 -04:00
//--- SHUTDOWN FLUSH: abandon an in-flight run instead of finishing it, and resume from the last
//--- COMPLETED, already-persisted era.
fix: flush the in-flight era on shutdown; sweep orphaned chart objects on attach Chart objects live in the MT5 chart PROFILE, not in this EA's files. They survive a terminal restart, a recompile, and deleting every .nnw/.cfg/.stats/.arrows on disk. Only a deinit that RUNS TO COMPLETION removes them - and MetaTrader force-terminates OnDeinit at roughly 4,500 ms, so a run killed mid-cleanup orphans them permanently with no owner left to clean up after. That is the "deleted every file, recompiled, restarted, old arrows and a stale panel still there" report: nothing was wrong with the files and deleting them could not have helped. Both halves are fixed. STOP OVERRUNNING THE BUDGET. OnDeinit used to finalise the in-flight run (StopTraining -> FinalizeTrainRun: checkpoint restore, live-state re-seed) and then write two full nets per chart. On four charts that is the bulk of the budget, spent to preserve a PARTIAL era that was never scored, never checkpointed and never deployable. FlushTrainRun() discards it instead - drop the resumable bookkeeping, leave the net neutral (unfreeze BN, flush the batch, batch size 1), skip the save - and training resumes from the last completed era, which the era-end save and the periodic autosave have already put on disk. What is discarded is bounded by one era. A CONVERGED model keeps the old finalise-and-save path: its weights can carry online-learning updates made since the last era boundary, and for a deployed model no further era boundary is coming to persist them. MAKE CLEANUP SELF-HEALING. Every purge sat behind a branch - no model loaded, sidecar missing - so the common paths returned leaving whatever the previous instance stranded. LoadChartSignals now sweeps the arrow namespace unconditionally before restoring, so the post-init chart holds exactly what the sidecar holds whichever branch runs, and the panel gets the same treatment before Create() (CAppDialog namespaces its controls, so a killed Destroy strands the lot and the next attach draws a second panel on the corpse). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:08:40 -04:00
bool FlushTrainRun(void)
{
bool inFlight = (m_trainRunActive || m_eraResumePending || m_labelPrebuildActive || m_simOosRunActive);
m_trainingStopRequested = true;
m_trainingPaused = false;
//--- Drop the resumable bookkeeping WITHOUT calling FinalizeTrainRun: no checkpoint restore, no
//--- persist, no dtStudied advance. The next start re-derives all of it from the saved model.
m_trainRunActive = false;
m_eraResumePending = false;
m_haveOosCheckpoint = false;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
m_checkpointEra = -1; // the joint-checkpoint era stamp goes with the snapshot it describes
fix: flush the in-flight era on shutdown; sweep orphaned chart objects on attach Chart objects live in the MT5 chart PROFILE, not in this EA's files. They survive a terminal restart, a recompile, and deleting every .nnw/.cfg/.stats/.arrows on disk. Only a deinit that RUNS TO COMPLETION removes them - and MetaTrader force-terminates OnDeinit at roughly 4,500 ms, so a run killed mid-cleanup orphans them permanently with no owner left to clean up after. That is the "deleted every file, recompiled, restarted, old arrows and a stale panel still there" report: nothing was wrong with the files and deleting them could not have helped. Both halves are fixed. STOP OVERRUNNING THE BUDGET. OnDeinit used to finalise the in-flight run (StopTraining -> FinalizeTrainRun: checkpoint restore, live-state re-seed) and then write two full nets per chart. On four charts that is the bulk of the budget, spent to preserve a PARTIAL era that was never scored, never checkpointed and never deployable. FlushTrainRun() discards it instead - drop the resumable bookkeeping, leave the net neutral (unfreeze BN, flush the batch, batch size 1), skip the save - and training resumes from the last completed era, which the era-end save and the periodic autosave have already put on disk. What is discarded is bounded by one era. A CONVERGED model keeps the old finalise-and-save path: its weights can carry online-learning updates made since the last era boundary, and for a deployed model no further era boundary is coming to persist them. MAKE CLEANUP SELF-HEALING. Every purge sat behind a branch - no model loaded, sidecar missing - so the common paths returned leaving whatever the previous instance stranded. LoadChartSignals now sweeps the arrow namespace unconditionally before restoring, so the post-init chart holds exactly what the sidecar holds whichever branch runs, and the panel gets the same treatment before Create() (CAppDialog namespaces its controls, so a killed Destroy strands the lot and the next attach draws a second panel on the corpse). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:08:40 -04:00
m_labelPrebuildActive = false;
if(m_simOosRunActive)
{
delete m_simOosNet;
m_simOosNet = NULL;
m_simOosRunActive = false;
}
//--- Leave the net in the same neutral state FinalizeTrainRun leaves it in - a frozen batch-norm or
//--- a half-filled mini-batch must not be what a later inference path finds. Cheap, unlike the save.
if(CheckPointer(Net) != POINTER_INVALID)
{
Net.SetBatchNormFrozen(false);
Net.FlushBatch();
Net.SetBatchSize(1);
}
return inFlight;
}
void StopTraining(void)
{
m_trainingStopRequested = true;
m_trainingPaused = false;
//--- ScheduleTrainingIfNeeded() refuses to schedule another "New Bar" event while
//--- m_trainingStopRequested is set, so a run interrupted mid-chunk would otherwise never get
//--- called again to finalize (restore the best checkpoint, persist state) - do it
//--- synchronously here instead.
if(m_trainRunActive)
FinalizeTrainRun();
Print(ID + ": training stopped by user (era " + IntegerToString(m_eraCount) + ", weights as of last completed era retained)");
diag: inference-path census, to explain zero-trade backtests A backtest of the CONVERGED CONV model produced "Final directional result: 0.00000000" on every one of 1744 bars and therefore zero trades. Nothing in the log could separate the three candidate causes, and each needs a different fix: 1. RefreshLatestSignal never called (new-bar gate never fires) 2. called, but bailing at one of its two early returns 3. running fine, and the model genuinely answers Neutral every bar Counts all three plus the Buy/Sell/Neutral split, printed once at shutdown via StopTraining (which the tester reaches through OnDeinit). Three increments per bar against a full feedForward - not worth gating. Ruled out while writing this, so the next session does not re-derive it: - the alternation gate (m_lastNonNeutralSignal) is NOT the cause. It starts at Neutral, so a first Buy would still fire and show up as one non-zero direction. We saw zero. It IS still a live hazard for a one-sided model - CONV currently calls Buy:17% Sell:0%, and after the first Buy every later Buy is suppressed until a Sell that never comes - but it cannot explain an all-zero run. - shallow buffers do not hard-fail the feature builder: the swing-context Donchian loop breaks gracefully when it runs off loaded history. It does mean converged-path inference computes Donchian/return/SMA features over a TRUNCATED window versus training, which is a real train/inference skew worth its own fix, but it degrades features rather than zeroing them. Both builds 0/0. Diagnostic only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 18:24:32 -04:00
PrintInferenceTally();
}
//--- Inference-path census, printed at shutdown. Each implies a completely different fix.
//--- Counting is the cheapest way to tell them apart and it costs nothing per bar.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
void NoteVoteGate(bool directional)
{
if(!directional)
return;
bool open = m_trainingComplete || (m_inferenceOnly && m_modelLoadedFromDisk);
if(m_voteGateCompleteAtFirst < 0)
{
m_voteGateCompleteAtFirst = (int)m_trainingComplete;
m_voteGateLoadedAtFirst = (int)m_modelLoadedFromDisk;
}
if(open)
m_voteGatePassed++;
else
m_voteGateBlocked++;
}
diag: inference-path census, to explain zero-trade backtests A backtest of the CONVERGED CONV model produced "Final directional result: 0.00000000" on every one of 1744 bars and therefore zero trades. Nothing in the log could separate the three candidate causes, and each needs a different fix: 1. RefreshLatestSignal never called (new-bar gate never fires) 2. called, but bailing at one of its two early returns 3. running fine, and the model genuinely answers Neutral every bar Counts all three plus the Buy/Sell/Neutral split, printed once at shutdown via StopTraining (which the tester reaches through OnDeinit). Three increments per bar against a full feedForward - not worth gating. Ruled out while writing this, so the next session does not re-derive it: - the alternation gate (m_lastNonNeutralSignal) is NOT the cause. It starts at Neutral, so a first Buy would still fire and show up as one non-zero direction. We saw zero. It IS still a live hazard for a one-sided model - CONV currently calls Buy:17% Sell:0%, and after the first Buy every later Buy is suppressed until a Sell that never comes - but it cannot explain an all-zero run. - shallow buffers do not hard-fail the feature builder: the swing-context Donchian loop breaks gracefully when it runs off loaded history. It does mean converged-path inference computes Donchian/return/SMA features over a TRUNCATED window versus training, which is a real train/inference skew worth its own fix, but it degrades features rather than zeroing them. Both builds 0/0. Diagnostic only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 18:24:32 -04:00
void PrintInferenceTally(void)
{
long attempts = m_refreshOk + m_refreshFailFeatures + m_refreshFailShort;
if(attempts <= 0)
{
Print(ID + ": inference census - RefreshLatestSignal was NEVER CALLED (0 attempts). The new-bar gate never fired.");
return;
}
Print(ID + ": inference census - ", attempts, " refresh attempts: ", m_refreshOk, " completed, ",
m_refreshFailFeatures, " bailed in BufferTempData, ", m_refreshFailShort, " bailed on a short feature window",
" | decisions Buy:", m_refreshBuy, " Sell:", m_refreshSell, " Neutral:", m_refreshNeutral);
//--- Second half of the census, and the half that separates "the model said nothing" from
//--- "the model spoke and was not allowed to vote" - see m_voteGateBlocked for why that
//--- distinction is the whole point.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
if(m_voteGateCompleteAtFirst < 0)
Print(ID + ": inference census - vote gate was NEVER REACHED (no directional decision ever hit "
"LongCondition/ShortCondition). Either every decision was Neutral, or this filter was never polled.");
else
Print(ID + ": inference census - vote gate passed:", m_voteGatePassed, " blocked:", m_voteGateBlocked,
" | at first vote trainingComplete=", (m_voteGateCompleteAtFirst != 0 ? "true" : "false"),
" modelLoadedFromDisk=", (m_voteGateLoadedAtFirst != 0 ? "true" : "false"),
" inferenceOnly=", (m_inferenceOnly ? "true" : "false"),
(m_voteGateBlocked > 0 && m_voteGatePassed == 0
? " <-- EVERY directional call was discarded here. This is the zero-direction cause."
: ""));
}
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
//--- The ONLY place the study event is posted: arms bEventStudy with THIS instance's id (so the
//--- handler in OnChartEventHandler(), which matches on m_studyEventId, is the only member that
//--- runs it) and stamps the lost-event watchdog. sparam tags ("New Bar"/"Init"/"Resume"/...) are
//--- purely diagnostic.
bool ArmStudyEvent(const long lparam, const string tag)
{
bEventStudy = EventChartCustom(ChartID(), m_studyEventId, lparam, 0, tag);
if(bEventStudy)
m_studyArmedTick = GetTickCount();
return bEventStudy;
}
void StartTraining(void)
{
if(!m_trainingStopRequested && !m_trainingPaused)
return;
m_trainingStopRequested = false;
m_trainingPaused = false;
if(!bEventStudy)
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
ArmStudyEvent((long)dtStudied, "Resume");
Print(ID + ": training (re)started by user (era " + IntegerToString(m_eraCount) + ")");
}
//--- Has an era ever cleared the per-class recall floor and been checkpointed this run? This is the
//--- same quality bar the plateau ladder's auto-deploy requires (see PLATEAU_STAGE_DEPLOY), exposed so
//--- the panel can warn before a MANUAL deploy ships a model that ignores Buy or Sell.
bool HasRecallPassingCheckpoint(void) const { return m_bestPassedRecall; }
//--- MANUAL deploy (panel "Deploy Model"): finalise whatever the run has found so far as THE
//--- model - exactly what the plateau ladder does on its own at stage 3, just triggered early by
//--- the operator. Reversible via RetrainDeployed().
bool DeployNow(void)
{
if(CheckPointer(Net) == POINTER_INVALID || !m_isInitialized)
return false;
if(m_trainingComplete)
return true; // already deployed - nothing to do
//--- Set BEFORE any save below: the flag is written INTO the .nnw, so persisting first would store
//--- "still training" and a restart would resume the era loop instead of running the deployed model.
m_trainingComplete = true;
m_trainingPaused = false;
m_trainingStopRequested = false;
if(m_trainRunActive || m_haveOosCheckpoint)
FinalizeTrainRun(); // restores the best checkpoint, persists, ends the run
else
{
//--- Nothing trained this session (e.g. deploying a model that was just loaded from disk), so
//--- there is no in-memory checkpoint to restore - persist exactly what is loaded right now.
PersistDeployedModel();
SaveChartSignals();
}
RefreshLatestSignal();
Print(ID + ": model DEPLOYED by user at era " + IntegerToString(m_eraCount) +
" (balanced accuracy " + (m_bestBalancedOos < 0 ? "n/a" : DoubleToString(m_bestBalancedOos, 1) + "%") +
", blended OOS " + DoubleToString(dOosForecast, 1) + "%) - training stopped, now running live inference" +
(m_enableOnlineLearning ? " with online continual learning" : "") +
". Use the panel's \"Retrain Model\" to resume training from here.");
//--- Deliberately reported, not enforced: a manual deploy is the operator overriding the
//--- ladder, and that override stays available. See DEPLOY_FAMILY_WISE_ALPHA and
//--- HasRecallPassingCheckpoint()'s panel warning.
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
ReportSelectionGateVerdict("manual deploy");
return true;
}
//--- The inverse of DeployNow(), and the ONLY way back: while m_trainingComplete is set,
//--- ScheduleTrainingIfNeeded() routes every tick to the converged/inference branch, so
//--- StartTraining() alone can never revive a deployed model (it clears the stop flag, but the
//--- complete flag still wins that branch).
void RetrainDeployed(void)
{
if(!m_trainingComplete)
return;
m_trainingComplete = false;
m_trainingStopRequested = false;
m_trainingPaused = false;
//--- Persist the cleared flag immediately. Otherwise a terminal restart before the first era
//--- completes would reload the .nnw still marked complete and silently go back to inference-only,
//--- looking like the button did nothing.
PersistDeployedModel();
if(!bEventStudy)
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
ArmStudyEvent((long)dtStudied, "Retrain");
Print(ID + ": RETRAINING the deployed model from era " + IntegerToString(m_eraCount) +
" - keeping its current weights as the starting point (use \"Delete & Reset Weights\" to start from scratch instead).");
}
//--- Manual "rescan" of the drawn signal arrows: purges every arrow currently on the chart (namespaced
//--- delete - user drawings untouched) and re-infers the last SIGNAL_RESCAN_LOOKBACK_BARS bars from the
//--- CURRENTLY deployed weights, then re-runs the same end-of-era NMS declutter (PruneDirectionalClusters)
//--- used during training so the fresh set matches what a live re-render would have produced. Wired to
//--- the panel's Hide->Show Signals sequence: without this, "restore" only ever replays whatever was
//--- last saved to the .arrows sidecar, which for a long-deployed model can be a stale historical render
//--- from whenever it was last actually trained - years-old arrows crowding out anything recent. Chart-only
//--- (no persistent chart in the tester/optimizer) and a no-op until a model has something to infer with.
//--- This only does the cheap setup (buffer resize, arrow purge, cache alloc) and QUEUES the per-bar
//--- inference loop for AdvanceChartSignalRescan() to drain in time-boxed slices off the timer - see
//--- that method's comment for why the loop itself must never run in one blocking pass. Returns true
//--- once a rescan has been queued (check RescanPending() for completion), false if there was nothing
//--- to rescan (no deployed model, tester/optimizer context, etc).
bool StartChartSignalRescan(void)
{
if(MQLInfoInteger(MQL_TESTER) || MQLInfoInteger(MQL_OPTIMIZATION) || MQLInfoInteger(MQL_FORWARD))
return false;
if(CheckPointer(Net) == POINTER_INVALID || !m_isInitialized || !m_trainingComplete)
return false;
if(m_outputNeuronsCount != 1 && m_outputNeuronsCount != 3)
return false;
int barsAvail = Bars(m_symbol.Name(), PERIOD_CURRENT);
int barsNow = MathMin(SIGNAL_RESCAN_LOOKBACK_BARS, barsAvail);
fix(depth): route EVERY ResizeBuffers call site through one indicator-depth gate 1dda479 clamped the training sweep. It left five other paths asking the indicators for a depth they cannot serve, and on a live account the quiet ones are worse than the stall was - a stalled chart is visible, a chart trading on a degraded feature window is not. ServableBars(want, context) is now the single gate, and all six go through it: training sweep clamp, floored at TRAIN_MIN_CLAMPED_BARS (below that a small positive BarsCalculated is warm-up, which m_coldSweepTick owns) label prebuild clamp - labels come from price/ADZigZag and would survive a capped MA, but ResizeBuffers sizes EVERY buffer and a failed CopyBuffer leaves m_MA EMPTY for the next reader, so this path could silently re-break the block Train()'s clamp just fixed live inference HOLD. Below `need` the swing block takes its degraded path and inference runs on a different feature distribution than the model was fitted on. This EA sizes real positions off that output, so no signal beats a mismatched one online learning HOLD, same reason and worse - this path WRITES to a live trading model, so a mismatched (features, label) pair is not a wrong arrow, it is a wrong weight update that compounds every bar chart rescan clamp - SIGNAL_RESCAN_LOOKBACK_BARS is 5000 and MT5's smallest "Max bars in chart" is also 5000, so this one is genuinely reachable; uncapped it repaints the window all-Neutral research export clamp before the emptiness test, so a capped symbol exports the depth it has rather than writing a CSV with a dead feature block - an artefact that looks complete and is silently wrong Both HOLDs are insurance, not expected states: `need` tops out near 1,152 bars (16 + 750 + 384 + 2) against a 5,000 floor on the terminal setting. They exist so the failure mode is unreachable rather than merely unlikely. Not changed: a genuinely SHORT price history still takes the old degraded path at every site. That is pre-existing behaviour and narrowing it would mute charts that trade today, so it stays a separate decision rather than a side effect of this fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 13:33:16 -04:00
//--- SIGNAL_RESCAN_LOOKBACK_BARS is 5000 and the SMALLEST "Max bars in chart" setting MT5 offers
//--- is also 5000, so this call site genuinely can be capped - it would redraw the whole rescan
//--- window as Neutral (every feature window rejected) and read as "the model calls nothing".
barsNow = ServableBars(barsNow, "chart rescan");
if(barsNow <= m_historyBars)
return false;
if(!ResizeBuffers(barsNow) || !RefreshData())
return false;
EnsureShadowNet();
//--- Drop only the arrows THIS rescan is about to re-judge - i.e. those within [now, oldest
//--- bar of the barsNow window] - not every namespaced arrow on the chart. This scoped delete
//--- is the fix - older arrows are never in scope to be wiped in the first place.
datetime rescanCutoffTime = m_Time.GetData(barsNow - 1);
feat(ui): thresholds pick from a dropdown, and the finder arrows are back beside the level lines Two UX changes the operator asked for. THRESHOLDS. Signal_ThresholdOpen/Close were raw ints with the legal range written in the label ("[0...100, 101 = never]") - the one input style this codebase converted away from everywhere else. Open now takes the existing PERCENTAGE_PRESETS, whose comment already declared itself to be "Signal_ ThresholdOpen's scale" but was never wired to it; Close takes a new SIGNAL_CLOSE_PRESETS carrying the same rungs plus CLOSE_DISABLED = 101, which is why it cannot just reuse the other enum. Member names are prefixed because MQL5 enum members share ONE flat namespace - a bare PCT_25 in the second enum would silently resolve to the first one's, warning only. Values are unchanged, so existing .set files keep their settings. Both call sites now cast explicitly at the CExpertSignal boundary rather than leaning on an implicit enum-to-int conversion that only warns. ARROWS. 2026-08-19 replaced the low/high arrows WITH trigger-price lines; that was a swap where it should have been an addition, and it cost the zoomed-out view. A mark is now both objects: the line is the precise entry/exit level, the arrow off the candle's extreme is the finder that says there is something here to zoom into. The arrow's name is the line's plus a suffix, so it stays inside SIG_ARROW_PREFIX and every prefix-scoped purge already reaches it. The two type-filtered sweeps had to widen or they would clear one half and leave the other: the Hide/Show visibility loop and the pre-rescan scoped delete both walked OBJ_TREND only. Both are typed-blind and prefix-scoped now - the same widening this file's 2026-08-09 note describes, for the same reason it gives. Deletes go through one WarriorDeleteSignalMark() so an arrow cannot outlive the line it belongs to, and the sidecar deliberately still records one row per mark off the line (the half carrying the price), with the restore redrawing the pair. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 13:50:33 -04:00
//--- TYPED-BLIND, PREFIX-SCOPED. A mark is a line AND an arrow since 2026-08-20, and a typed
//--- scan clears only the half it names - leaving stale arrows on bars the rescan no longer
//--- signals. The prefix test is what keeps a typed-blind sweep off the user's own drawings.
for(int oi = ObjectsTotal(0, -1, -1) - 1; oi >= 0; oi--)
{
feat(ui): thresholds pick from a dropdown, and the finder arrows are back beside the level lines Two UX changes the operator asked for. THRESHOLDS. Signal_ThresholdOpen/Close were raw ints with the legal range written in the label ("[0...100, 101 = never]") - the one input style this codebase converted away from everywhere else. Open now takes the existing PERCENTAGE_PRESETS, whose comment already declared itself to be "Signal_ ThresholdOpen's scale" but was never wired to it; Close takes a new SIGNAL_CLOSE_PRESETS carrying the same rungs plus CLOSE_DISABLED = 101, which is why it cannot just reuse the other enum. Member names are prefixed because MQL5 enum members share ONE flat namespace - a bare PCT_25 in the second enum would silently resolve to the first one's, warning only. Values are unchanged, so existing .set files keep their settings. Both call sites now cast explicitly at the CExpertSignal boundary rather than leaning on an implicit enum-to-int conversion that only warns. ARROWS. 2026-08-19 replaced the low/high arrows WITH trigger-price lines; that was a swap where it should have been an addition, and it cost the zoomed-out view. A mark is now both objects: the line is the precise entry/exit level, the arrow off the candle's extreme is the finder that says there is something here to zoom into. The arrow's name is the line's plus a suffix, so it stays inside SIG_ARROW_PREFIX and every prefix-scoped purge already reaches it. The two type-filtered sweeps had to widen or they would clear one half and leave the other: the Hide/Show visibility loop and the pre-rescan scoped delete both walked OBJ_TREND only. Both are typed-blind and prefix-scoped now - the same widening this file's 2026-08-09 note describes, for the same reason it gives. Deletes go through one WarriorDeleteSignalMark() so an arrow cannot outlive the line it belongs to, and the sidecar deliberately still records one row per mark off the line (the half carrying the price), with the restore redrawing the pair. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 13:50:33 -04:00
string onm = ObjectName(0, oi, -1, -1);
if(StringFind(onm, SIG_ARROW_PREFIX) != 0)
continue;
if((datetime)ObjectGetInteger(0, onm, OBJPROP_TIME) >= rescanCutoffTime)
ObjectDelete(0, onm);
}
ArrayResize(m_arrowSignalCache, barsNow);
ArrayInitialize(m_arrowSignalCache, -2.0);
m_rescanBarsNow = barsNow;
m_rescanHi = barsNow - m_historyBars;
m_rescanIndex = 0;
m_rescanRawBuy = 0;
m_rescanRawSell = 0;
m_rescanRawNeutral = 0;
m_rescanPending = (m_rescanHi > 0);
m_rescanStartMs = GetTickCount();
if(m_rescanPending)
Print(ID + ": rescanning last " + IntegerToString(m_rescanHi) + " bars against the deployed model (progressive, non-blocking)...");
return m_rescanPending;
}
//--- true while a queued rescan (StartChartSignalRescan above) still has slices left for
//--- AdvanceChartSignalRescan to drain - polled by Warrior_EA.mq5's FinalizeSignalsRescanIfDone() to
//--- know when it's safe to (re)apply arrow visibility and report the Show Signals click as complete.
bool RescanPending(void) const { return m_rescanPending; }
//--- forces a save of the network's current in-memory weights/state regardless of era-completion
//--- state; called from OnDeinit() so shutdown/chart-removal never loses more than the current
//--- tick of learning, and a subsequent restart's Train() resumes from m_eraCount rather than
//--- the last fully-completed era only.
bool PersistWeightsOnShutdown(void)
{
if(CheckPointer(Net) == POINTER_INVALID || !m_isInitialized)
return false;
//--- An inference-only run (any Strategy Tester pass - see m_inferenceOnly) trains NOTHING,
//--- so there is no new state to persist and this save can only do harm.
if(m_inferenceOnly)
{
PrintVerbose(ID + ": inference-only run - skipping the shutdown weight save (nothing was trained; the cached model is left exactly as seeded).");
return true;
}
//--- Nothing trained and nothing loaded => there is no state to persist, and writing anyway
//--- is actively harmful.
fix(ai): stop the shutdown save from resurrecting reset weights; size HYBRID's LSTM to its real fan-in ResetWeights already deletes the whole model set - .nnw, .cfg, _ckpt.tmp, .stats, _shadow.nnw - and clears both the .arrows sidecar and the drawn chart objects. What undid it was PersistWeightsOnShutdown: detaching the EA after a reset but before an era completed re-created a .nnw from the freshly-built, never-run net, so the next attach loaded an era-0 stub instead of starting clean. For LSTM/HYBRID that stub is worse than nothing - a layer that has never run a forward pass has m_iInputs<=0, so Save omits every LSTM buffer (see 413ff7e). Skip the save when no era completed and no model was loaded; that is exactly the post-reset and first-attach state. Also sweep _shadowclone.tmp, which the reset did not cover. Separately, ComputeLstmHiddenSize budgeted every topology against the flattened input (historyBars x neuronsCount). True for LSTM, wrong for HYBRID, where AddConvStage runs first and the LSTM is fed the conv feature map - historyBars x convFilterCount, 160 rather than 420 at H1 defaults. The quadratic is dominated by the inputs term, so overstating the fan-in 2.6x cost a full ladder step (16 units where the budget affords 32). New virtual HasConvBeforeLstm() feeds LstmFanIn(), so composition decides this rather than an AIType check. desc.window is advisory only - CNet never passes it to the layer - but is now truthful for the same reason. Derived values stay out of the weights-filename fingerprint and are adopted from the .cfg, so existing models keep their saved width; only fresh ones pick up the corrected budget. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 13:06:09 -04:00
if(m_eraCount == 0 && !m_modelLoadedFromDisk)
{
PrintVerbose(ID + ": no era completed and no model loaded - skipping the shutdown weight save (leaving the model files absent so the next attach starts genuinely clean).");
return true;
}
double currentIndicatorParams[];
m_indicatorTuner.Flatten(currentIndicatorParams);
bool ok = Net.Save(m_activeFileName + ".nnw", dError, dUndefine, dForecast, dtStudied, m_activeFileCommon, m_eraCount, m_trainingComplete, currentIndicatorParams);
//--- calibration state (class priors + confidence scale) must travel with the weights so live
//--- trading behaves like training after a restart - see SaveModelStats().
if(!SaveModelStats(m_activeFileName, m_activeFileCommon))
Print(ID + ": ERROR - shutdown SaveModelStats failed for " + m_activeFileName + ". Calibration state not persisted.");
//--- Deliberately do NOT save the shadow net here. Worst case a shutdown loses only the
//--- shadow's in-progress-era drift, which re-converges - a far better trade than risking the
//--- whole model to an over-budget shutdown.
if(!ok)
Print(ID + ": ERROR - failed to persist weights on shutdown for " + m_activeFileName + ", error " + IntegerToString(GetLastError()));
else
PrintVerbose(ID + ": weights persisted on shutdown (era " + IntegerToString(m_eraCount) + ", trainingComplete=" + (string)m_trainingComplete + ")");
return ok;
}
//--- Persist the drawn arrows to disk, then remove THIS EA's chart visuals (arrows + status
//--- label). Called early in OnDeinit(), before the heavy weight save, so a later stall/fault in
//--- the save can never leave the chart littered. Deliberately NOT part of SaveWeightsNow(): a
//--- mid-session manual save must not wipe the chart.
fix(chart): arrows survived the EA that drew them - persist, then clear Reported: on deinit the panel and status label go, the signal arrows stay. Two independent causes, both fixed here. 1. It was partly deliberate. ShutdownChartCleanup carried a second behaviour selected by a `preserveChartArrows` flag derived from the deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the arrows were left on the chart on purpose, to avoid a reload flicker. That branch IS the reported symptom, an operator cannot tell it apart from a cleanup that failed, and it was outright wrong whenever the reload changed the config - REASON_PARAMETERS means exactly that, and the preserved arrows then belonged to a model the chart no longer runs, with nothing marking them stale. It is gone, along with the flag and m_purgeChartOnDestruct. One path now: persist, clear, restore on the next attach. 2. Whatever remains was unfalsifiable. PurgeChart was a single ObjectsDeleteAll(prefix) whose return value was discarded, with no caller ever looking at the chart again - so "the arrows are still there" and "the arrows were never there" produced identical evidence, which is why the report survived three sessions. It now verifies: after the bulk delete it walks the OBJ_ARROW-typed list (a handful of objects, not the whole chart), deletes any surviving WarSig_ by name, and says so. Costs one typed scan when the bulk delete works, which is the normal case; names the root cause when it does not. Every failure mode of SaveChartSignals was also silent - it returned void and had three bare early returns. It returns bool now, logs the open error with the filename, and the shutdown purge is CONDITIONAL on it: for a converged model the chart objects are the only copy of its signal history (nothing redraws them - the renderer runs per training era and a deployed model has none left), so a chart left littered because the disk write failed beats a clean chart bought by destroying the history. Either way the log now says which happened. Also states the user's rule once, where arrows come back rather than across InitNeuralNetwork's several exits: no weights loaded for this config => clear the sidecar and start visually clean. A fresh run must not inherit calls it never made, and the first save would otherwise adopt them (the sidecar is rebuilt by scanning the chart). Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
void ShutdownChartCleanup(void)
{
fix(chart): arrows survived the EA that drew them - persist, then clear Reported: on deinit the panel and status label go, the signal arrows stay. Two independent causes, both fixed here. 1. It was partly deliberate. ShutdownChartCleanup carried a second behaviour selected by a `preserveChartArrows` flag derived from the deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the arrows were left on the chart on purpose, to avoid a reload flicker. That branch IS the reported symptom, an operator cannot tell it apart from a cleanup that failed, and it was outright wrong whenever the reload changed the config - REASON_PARAMETERS means exactly that, and the preserved arrows then belonged to a model the chart no longer runs, with nothing marking them stale. It is gone, along with the flag and m_purgeChartOnDestruct. One path now: persist, clear, restore on the next attach. 2. Whatever remains was unfalsifiable. PurgeChart was a single ObjectsDeleteAll(prefix) whose return value was discarded, with no caller ever looking at the chart again - so "the arrows are still there" and "the arrows were never there" produced identical evidence, which is why the report survived three sessions. It now verifies: after the bulk delete it walks the OBJ_ARROW-typed list (a handful of objects, not the whole chart), deletes any surviving WarSig_ by name, and says so. Costs one typed scan when the bulk delete works, which is the normal case; names the root cause when it does not. Every failure mode of SaveChartSignals was also silent - it returned void and had three bare early returns. It returns bool now, logs the open error with the filename, and the shutdown purge is CONDITIONAL on it: for a converged model the chart objects are the only copy of its signal history (nothing redraws them - the renderer runs per training era and a deployed model has none left), so a chart left littered because the disk write failed beats a clean chart bought by destroying the history. Either way the log now says which happened. Also states the user's rule once, where arrows come back rather than across InitNeuralNetwork's several exits: no weights loaded for this config => clear the sidecar and start visually clean. A fresh run must not inherit calls it never made, and the first save would otherwise adopt them (the sidecar is rebuilt by scanning the chart). Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
PersistAndClearChartSignals();
}
//--- Full shutdown persistence (weights + arrows), preserved for the panel's manual "save weights"
//--- button (SaveWeightsNow) - does NOT purge the chart. OnDeinit no longer calls this; it runs
//--- ShutdownChartCleanup() then PersistWeightsOnShutdown() so cleanup can't be starved by the save.
bool PersistOnShutdown(void)
{
bool ok = PersistWeightsOnShutdown();
//--- Persist the drawn arrows too so a re-add/recompile restores them without a retrain.
SaveChartSignals();
return ok;
}
//--- explicit manual save, identical persistence to PersistOnShutdown() but user-triggered from the panel
bool SaveWeightsNow(void) { return PersistOnShutdown(); }
//--- reloads this signal's current-config weights file from disk, discarding any unsaved in-memory
//--- training progress since the last successful save
bool LoadWeightsNow(void)
{
if(CheckPointer(Net) == POINTER_INVALID)
return false;
double loadedIndicatorParams[];
bool netLoaded = Net.Load(m_activeFileName + ".nnw", dError, dUndefine, dForecast, dtStudied, m_activeFileCommon, m_eraCount, m_trainingComplete, loadedIndicatorParams);
if(!netLoaded)
{
Print(ID + ": ERROR - failed to load weights from " + m_activeFileName + ".nnw, error " + IntegerToString(GetLastError()));
return false;
}
m_modelLoadedFromDisk = true;
fix: stop a .nnw from pinning a superseded architecture A .nnw persists the ARCHITECTURE, not just the weights: Save writes (int)activation per neuron and Load reads it straight back. The activation chosen in BuildFreshTopology() therefore only ever reached a brand-new topology - every reload restored the file's value and the next save wrote it back out, so a wrong value could never heal while the source read as though it were already fixed. That is how five models kept training with an unbounded NONE classification head for a full day after the 07-28 revert to SIGMOID. Confirmed by parsing the binaries: 848cb42c.nnw / 2e754b43.nnw carry `act=NONE` on the 3-neuron output layer, while a genuinely reset model of the same config carries act=SIGMOID. In the log it showed as negative "OOS raw out" values - impossible under sigmoid - escalating to a 4.14e13 logit spread with all three classes numerically identical (input-independent output) and balanced accuracy pinned on the 33.3% one-class floor. - OutputLayerActivation() is now the single source of truth, called by both BuildFreshTopology() and the new load-time repair, so the two can no longer diverge the way a duplicated literal did. - CNet::EnforceOutputActivation() re-asserts it after Load and reports the stale value; CExpertSignalAIBase::EnforceTopologyContract() logs the repair loudly, since weights learned under the old head may not be worth keeping even once the head is corrected. - Hidden layers are deliberately left alone: they legitimately differ per stage (PRELU dense/conv, NONE pool, TANH LSTM). Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 12:00:40 -04:00
//--- the file may carry a superseded architecture - correct it before anything reads the net
EnforceTopologyContract();
//--- restore the calibration state that pairs with these weights (priors + confidence scale) so a
//--- manual reload keeps live decisions calibrated exactly as the saved model was - see LoadModelStats().
LoadModelStats(m_activeFileName, m_activeFileCommon);
if(ArraySize(loadedIndicatorParams) == AD_TUNE_PARAM_COUNT)
{
2026-08-13 10:23:11 -04:00
//--- same no-change guard as the resume path - see AdoptIndicatorParams
if(m_indicatorsPtr != NULL)
2026-08-13 10:23:11 -04:00
AdoptIndicatorParams(loadedIndicatorParams, m_indicatorsPtr);
else
m_indicatorTuner.Unflatten(loadedIndicatorParams);
}
//--- Discard resumable state: the load just swapped dtStudied/m_eraCount/weights out from under
//--- whatever era a chunked Train() was mid-way through.
m_trainRunActive = false;
m_eraResumePending = false;
m_haveOosCheckpoint = false;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
m_checkpointEra = -1; // the joint-checkpoint era stamp goes with the snapshot it describes
m_oosWindow.Clear();
m_tuneTrialIndex = -1;
RefreshLatestSignal();
Print(ID + ": weights reloaded from disk (era " + IntegerToString(m_eraCount) + ", trainingComplete=" + (string)m_trainingComplete + ")");
return true;
}
//--- deletes this signal's current-config saved files only (weights, topology config, in-progress
//--- Deletes this config's saved files only and rebuilds a fresh untrained topology, so training
//--- restarts from era 0. m_fileName embeds symbol+period+id+outputs+algo, so no other model's
//--- files are reachable from here.
bool ResetWeights(void)
{
bool stopped = m_trainingStopRequested;
m_trainingStopRequested = true; // hold off any in-flight Train() scheduling while we reset
//--- Whichever file this run trains against (see InitNeuralNetwork): the shared production
//--- weights, or the tester cache during a backtest - so a panel reset mid-backtest cannot wipe
//--- the live model.
int flags = m_activeFileCommon ? FILE_COMMON : 0;
string nnw = m_activeFileName + ".nnw";
string cfg = m_activeFileName + ".cfg";
string ckpt = m_activeFileName + "_ckpt.tmp";
//--- Sidecars pair with the weights being erased: .stats carries the calibration, _shadow.nnw the
//--- deployed EMA, and _shadowclone.tmp is the clone staging file. Leave any behind and a fresh
//--- retrain inherits the OLD model's calibration or blends into a stale shadow.
string stats = m_activeFileName + ".stats";
string shadow = m_activeFileName + "_shadow.nnw";
fix(ai): stop the shutdown save from resurrecting reset weights; size HYBRID's LSTM to its real fan-in ResetWeights already deletes the whole model set - .nnw, .cfg, _ckpt.tmp, .stats, _shadow.nnw - and clears both the .arrows sidecar and the drawn chart objects. What undid it was PersistWeightsOnShutdown: detaching the EA after a reset but before an era completed re-created a .nnw from the freshly-built, never-run net, so the next attach loaded an era-0 stub instead of starting clean. For LSTM/HYBRID that stub is worse than nothing - a layer that has never run a forward pass has m_iInputs<=0, so Save omits every LSTM buffer (see 413ff7e). Skip the save when no era completed and no model was loaded; that is exactly the post-reset and first-attach state. Also sweep _shadowclone.tmp, which the reset did not cover. Separately, ComputeLstmHiddenSize budgeted every topology against the flattened input (historyBars x neuronsCount). True for LSTM, wrong for HYBRID, where AddConvStage runs first and the LSTM is fed the conv feature map - historyBars x convFilterCount, 160 rather than 420 at H1 defaults. The quadratic is dominated by the inputs term, so overstating the fan-in 2.6x cost a full ladder step (16 units where the budget affords 32). New virtual HasConvBeforeLstm() feeds LstmFanIn(), so composition decides this rather than an AIType check. desc.window is advisory only - CNet never passes it to the layer - but is now truthful for the same reason. Derived values stay out of the weights-filename fingerprint and are adopted from the .cfg, so existing models keep their saved width; only fresh ones pick up the corrected budget. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 13:06:09 -04:00
string shadowClone = m_activeFileName + "_shadowclone.tmp";
//--- SAY WHAT HAPPENED TO EVERY FILE. This once printed only on a delete FAILURE, so a four-member
//--- reset was 24 silent operations and "it only wiped the first model" could not be settled from
//--- a log. "absent" on a member that should have had a .nnw is a different fault from "deleted".
fix(reset): say what the reset actually did, per member and per file The user reports "Delete & Reset Weights only wipes the first NN". I could not find a code path that skips ensemble members, and I am not going to assert one: the handler loops g_aiSignals[0..g_aiSignalCount), all four topologies register unconditionally in OnInit, and SetIdentity gives each its own State\<id>\ folder so the six deleted paths are genuinely distinct per member. What IS true is that the whole success path was SILENT - six FileDelete calls per member printing only on failure, and one chart-wide Alert - so a four-member reset and a one-member reset produce byte-identical output. The symptom could be neither confirmed nor refuted from a log. That is the defect I can fix today. - COMPILED <timestamp> (__DATETIME__) beside the build tag. The hand-edited tag had sat at scan-nofwd-v5 across a week of commits, so it could not answer the question it exists for. The compile stamp cannot be forgotten. Tag bumped to reset-census-v6. - RegistryLine() (public): ID, active file path, common/local, era, deployed vs training, ensemble index. The reset handler prints a numbered census of the whole registry BEFORE the confirm dialog. If that says 1 on an AI_HYBRID chart the fault is registration, not the reset - and RegisterAISignal already has a loud MAX_AI_SIGNALS message for exactly that. - The confirmation dialog now names the count, so a wrong registry is visible before anything is deleted rather than after. - ResetWeights prints one line per member: N deleted / N already absent / N FAILED, plus a per-suffix breakdown. "absent" on a member that should have had a .nnw is a completely different fault from "deleted"; they were identical. - ResetWeights' return value was discarded. A member whose BuildFreshTopology fails has had its files deleted and has no network - and the Alert still said "weights reset". Counted now, with an INCOMPLETE alert when they disagree. - Same for dbm.ResetDatabase(), whose bool was also dropped. The DB is one shared file for every signal on the chart, so there is nothing per-member to loop - the log now says that explicitly, since it is the question being asked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 17:57:26 -04:00
int filesDeleted = 0, filesAbsent = 0, filesFailed = 0;
string wipeReport = "";
string targets[6];
targets[0] = nnw;
targets[1] = cfg;
targets[2] = ckpt;
targets[3] = stats;
targets[4] = shadow;
targets[5] = shadowClone;
for(int fi = 0; fi < 6; fi++)
{
ResetLastError();
string leaf = targets[fi];
string shortName = StringSubstr(targets[fi], StringLen(m_activeFileName)); // suffix only; full path prints below
fix(reset): say what the reset actually did, per member and per file The user reports "Delete & Reset Weights only wipes the first NN". I could not find a code path that skips ensemble members, and I am not going to assert one: the handler loops g_aiSignals[0..g_aiSignalCount), all four topologies register unconditionally in OnInit, and SetIdentity gives each its own State\<id>\ folder so the six deleted paths are genuinely distinct per member. What IS true is that the whole success path was SILENT - six FileDelete calls per member printing only on failure, and one chart-wide Alert - so a four-member reset and a one-member reset produce byte-identical output. The symptom could be neither confirmed nor refuted from a log. That is the defect I can fix today. - COMPILED <timestamp> (__DATETIME__) beside the build tag. The hand-edited tag had sat at scan-nofwd-v5 across a week of commits, so it could not answer the question it exists for. The compile stamp cannot be forgotten. Tag bumped to reset-census-v6. - RegistryLine() (public): ID, active file path, common/local, era, deployed vs training, ensemble index. The reset handler prints a numbered census of the whole registry BEFORE the confirm dialog. If that says 1 on an AI_HYBRID chart the fault is registration, not the reset - and RegisterAISignal already has a loud MAX_AI_SIGNALS message for exactly that. - The confirmation dialog now names the count, so a wrong registry is visible before anything is deleted rather than after. - ResetWeights prints one line per member: N deleted / N already absent / N FAILED, plus a per-suffix breakdown. "absent" on a member that should have had a .nnw is a completely different fault from "deleted"; they were identical. - ResetWeights' return value was discarded. A member whose BuildFreshTopology fails has had its files deleted and has no network - and the Alert still said "weights reset". Counted now, with an INCOMPLETE alert when they disagree. - Same for dbm.ResetDatabase(), whose bool was also dropped. The DB is one shared file for every signal on the chart, so there is nothing per-member to loop - the log now says that explicitly, since it is the question being asked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 17:57:26 -04:00
if(!FileIsExist(leaf, flags))
{
filesAbsent++;
wipeReport += StringFormat("%s%s=absent", (wipeReport == "" ? "" : " "), shortName);
continue;
}
if(FileDelete(leaf, flags))
{
filesDeleted++;
wipeReport += StringFormat("%s%s=deleted", (wipeReport == "" ? "" : " "), shortName);
}
else
{
filesFailed++;
wipeReport += StringFormat("%s%s=FAILED(%d)", (wipeReport == "" ? "" : " "),
shortName, GetLastError());
Print(ID + ": ERROR - failed to delete " + leaf + ", error " + IntegerToString(GetLastError()));
}
}
PrintFormat("%s: RESET WIPE of %s - %d deleted, %d already absent, %d FAILED | %s",
ID, m_activeFileName, filesDeleted, filesAbsent, filesFailed, wipeReport);
//--- The arrows and their .arrows sidecar belong to the model being erased, like the sidecars above.
fix: clear stale signal arrows when a fresh model starts at era 0 Arrow cleanup existed on two paths - the panel's reset-weights, and the topology-mismatch discard - but both are gated on there being a saved .nnw to delete. The third case had no cleanup at all: a fresh topology at era 0 with no weights behind it, which is what a changed config produces. A new fingerprint makes a new m_fileName, so the previous model's files are not "discarded", they are simply not this model's files, and nothing ever cleared the chart. That is not cosmetic. Arrows outlive the model that drew them twice over: 1. The chart objects live in the CHART, not the sidecar, so they survive a remove/re-add, a recompile, a restart and a fresh deploy no matter what happens to any file on disk. 2. SaveChartSignals() rebuilds the sidecar by SCANNING the chart for SIG_ARROW_PREFIX objects. So the first save of the fresh run adopts the dead model's calls and writes them out under the NEW model's filename - laundering them into the new model's history where nothing can separate them afterwards. Extracted the duplicated cleanup into ClearPersistedChartSignals(reason) - it cancels the deferred restore queue, deletes m_fileName + ".arrows", clears the namespaced chart objects and logs why - and called it from all three paths. The call sits at the BuildFreshTopology() call site, not inside it: the genetic tuner rebuilds a throwaway topology per candidate (AutoTune.mqh) and must never touch the chart. All three sites run after m_fileName has its config fingerprint appended, so they target the right sidecar. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 10:13:01 -04:00
ClearPersistedChartSignals("weights reset from the panel");
m_eraCount = 0;
m_trainingComplete = false;
m_modelLoadedFromDisk = false;
dtStudied = 0;
dError = -1;
dUndefine = 0;
dForecast = 0;
dPrevSignal = 0;
m_nmsLiveBuyTime = 0;
m_nmsLiveSellTime = 0;
m_nmsLiveBuyAccept = false;
m_nmsLiveSellAccept = false;
m_nmsLiveKeptTime = 0;
m_nmsLiveKeptDir = Neutral;
m_nmsLiveKeptConf = 0;
dOosError = -1;
dOosForecast = 0;
m_oosSamples = 0;
//--- Lifetime counters: reset ONLY here. A normal restart restores them from .stats.
m_cumIsCorrect = 0;
m_cumIsTotal = 0;
m_cumOosCorrect = 0;
m_cumOosTotal = 0;
if(m_ensembleMember)
{
g_ensCumOosCorrect = 0;
g_ensCumOosTotal = 0;
}
//--- In-progress chunked run/tuning state references buffers and checkpoints from before the reset.
m_trainRunActive = false;
m_eraResumePending = false;
m_haveOosCheckpoint = false;
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint The unit of evaluation in ensemble mode becomes the vote, because the vote is what trades (user: "at the end of the day they will vote together during live trading so that would make sense"). Four decisions move from the member to the ensemble: * which era is "best" -> the era whose COMBINED VOTE scored best * what is checkpointed -> a JOINT snapshot: every member's weights at that one era * when the run gives up -> one shared plateau ladder * whether it may deploy -> family-wise gate on the vote WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each net's own best era, and those eras differ. The resulting quartet was never measured together at any instant, so the vote it casts live is a configuration no OOS number ever described. Capturing all four at the era whose vote won makes the deployed ensemble exactly the measured one. Correct because of the era barrier (b77e7b4): Train() runs at most one era per call and a member that finished era N is held until every member reaches N, so when the last member scores the vote no member's weights have advanced past end-of-era-N. That makes the deferred simultaneous capture a guarantee rather than a race. Each snapshot is era-STAMPED and deploy requires every stamp to equal the winning era - otherwise a member whose capture failed would still hold an older snapshot and the deployed quartet would again be one nothing measured. Partial capture rolls the era back out of "best" so the search continues instead of freezing behind a checkpoint that does not exist. Statistics mirror the per-member gate one for one - same coverage floor (MIN_COVERAGE_FRACTION_OF_BASE_RATE), same always-call-one-direction chance reference, same EDGE_MIN_SIGMAS margin, same Sidak correction over the eras ranked (DEPLOY_FAMILY_WISE_ALPHA). Only the population differs: the bars the VOTE fired on, at Min_Vote_Open, rather than the bars one member called. Two-sidedness is required of the vote itself - a vote that never goes short IS the always-long model the chance reference prices in. Members keep their own per-era statistics and their own learning-rate dynamics (regression restore, eta decay); those are per-net training mechanics, not deployment decisions. The shared ladder is mirrored onto each member so per-era log lines report the state that actually governs them. Solo charts are untouched on every path. Verified: full MetaEditor compile, 0 errors 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:25:25 -04:00
m_checkpointEra = -1; // the joint-checkpoint era stamp goes with the snapshot it describes
m_oosWindow.Clear();
m_syncWaitStartTick = 0;
m_tuneTrialIndex = -1;
//--- Re-verify history sync and rebuild the label cache: both reference bars from before the reset.
m_warmupPassesRemaining = 3;
m_labelCacheBars = 0;
m_labelCacheAnchorTime = 0;
m_labelCachePrebuilt = false;
m_labelPrebuildActive = false;
m_prebuildSeedPending = false;
//--- A reset declares there are no fitted weights left to protect, so the geometry must
//--- re-derive. m_geometryDerivePasses is the one that blocks it: LoadAndCompare pins it to
//--- BARRIER_DERIVE_MAX_PASSES so an EXISTING model can never move its target, which is right
//--- for a load and wrong here. Miss this and every reset relabels under the OLD pair.
fix(geometry): a weights reset could never change the barrier - the pair laundered itself through the wipe Reported as "it still seems leaned towards 2x atr" after a full Delete && Reset Weights on SP500 H4. It was not the .cfg pin, and it was not the new ratio floor failing to take: the geometry is held in members ResetWeights never cleared, so a reset wiped the .cfg, built a fresh net, restarted at era 0 - and then relabelled under the PREVIOUS model's pair, before SaveTopologyConfiguration wrote that stale pair back into the brand new .cfg. Reset, re-derive, re-pin, with the middle step missing. No number of resets could ever have moved it. Evidence in the 2026-08-19 journal: RESET WIPE at 16:23:38.699 (.cfg=deleted), "rebuilt a fresh topology" at .790, and a label cache at .890 with a distribution byte-identical to the pre-reset 2.00/2.00 one (Buy 3221 / Sell 3303 / Neutral 4837, mean lifespan 6.4 bars) - 100 ms later, with no DeriveBarrierGeometry between them. The member that actually blocked it is m_geometryDerivePasses. LoadAndCompareTopologyConfiguration pins it to BARRIER_DERIVE_MAX_PASSES to block the fixed-point iteration, which is correct for a LOAD - an existing model must never re-derive or the target moves under fitted weights - and exactly wrong for a RESET, which is the act of declaring there are no fitted weights left to protect. State that is correctly sticky for one lifecycle event, silently inherited by another; the same shape as the .cfg pin sitting beside it. ResetWeights now returns the whole derivation to its constructor state: the three latches, the pair, the horizon and its flags, the swing/lifespan measurements, the scan's own outputs, and m_spreadAtr - that last one matters because the cost filter is deliberately inert on pass 1 (m_spreadAtr <= 0.0) and an inherited spread makes a reset model walk a different ladder than a genuinely new one. m_sl_mode/m_tp_mode go back to the SL_Mode/TP_Mode inputs, since the adopt path overwrites them in place with no copy of what was configured. NOT COMPILED - user compiles in MetaEditor.
2026-08-19 16:31:18 -04:00
m_geometryDerived = false;
m_geometryAdopted = false;
m_geometryCfgSaved = false; // let the re-derived pair pin itself to the fresh .cfg
m_geometryDerivePasses = 0; // the fixed-point iteration runs again from scratch
m_derivedSlMult = 0.0;
m_derivedTpMult = 0.0;
//--- These feed the derivation, so they re-measure with it - a stale horizon would size the new
//--- target from travel measured under the old one.
fix(geometry): a weights reset could never change the barrier - the pair laundered itself through the wipe Reported as "it still seems leaned towards 2x atr" after a full Delete && Reset Weights on SP500 H4. It was not the .cfg pin, and it was not the new ratio floor failing to take: the geometry is held in members ResetWeights never cleared, so a reset wiped the .cfg, built a fresh net, restarted at era 0 - and then relabelled under the PREVIOUS model's pair, before SaveTopologyConfiguration wrote that stale pair back into the brand new .cfg. Reset, re-derive, re-pin, with the middle step missing. No number of resets could ever have moved it. Evidence in the 2026-08-19 journal: RESET WIPE at 16:23:38.699 (.cfg=deleted), "rebuilt a fresh topology" at .790, and a label cache at .890 with a distribution byte-identical to the pre-reset 2.00/2.00 one (Buy 3221 / Sell 3303 / Neutral 4837, mean lifespan 6.4 bars) - 100 ms later, with no DeriveBarrierGeometry between them. The member that actually blocked it is m_geometryDerivePasses. LoadAndCompareTopologyConfiguration pins it to BARRIER_DERIVE_MAX_PASSES to block the fixed-point iteration, which is correct for a LOAD - an existing model must never re-derive or the target moves under fitted weights - and exactly wrong for a RESET, which is the act of declaring there are no fitted weights left to protect. State that is correctly sticky for one lifecycle event, silently inherited by another; the same shape as the .cfg pin sitting beside it. ResetWeights now returns the whole derivation to its constructor state: the three latches, the pair, the horizon and its flags, the swing/lifespan measurements, the scan's own outputs, and m_spreadAtr - that last one matters because the cost filter is deliberately inert on pass 1 (m_spreadAtr <= 0.0) and an inherited spread makes a reset model walk a different ladder than a genuinely new one. m_sl_mode/m_tp_mode go back to the SL_Mode/TP_Mode inputs, since the adopt path overwrites them in place with no copy of what was configured. NOT COMPILED - user compiles in MetaEditor.
2026-08-19 16:31:18 -04:00
m_barrierHorizonResolved = false;
m_barrierHorizonBars = BARRIER_HORIZON_FALLBACK;
m_barrierHorizonLegStarved = false;
m_barrierHorizonClamped = false;
m_barrierFallbackWarned = false;
m_swingMedianLegAtr = 0.0;
m_lastRungLifespan = 0.0;
//--- Back to 0, not merely left alone: DeriveBarrierGeometry's cost filter is deliberately inert
//--- on pass 1 (m_spreadAtr <= 0), and an inherited value makes it reject rungs on a cost it was
//--- designed not to know yet - a different ladder than a genuinely new model would walk.
fix(geometry): a weights reset could never change the barrier - the pair laundered itself through the wipe Reported as "it still seems leaned towards 2x atr" after a full Delete && Reset Weights on SP500 H4. It was not the .cfg pin, and it was not the new ratio floor failing to take: the geometry is held in members ResetWeights never cleared, so a reset wiped the .cfg, built a fresh net, restarted at era 0 - and then relabelled under the PREVIOUS model's pair, before SaveTopologyConfiguration wrote that stale pair back into the brand new .cfg. Reset, re-derive, re-pin, with the middle step missing. No number of resets could ever have moved it. Evidence in the 2026-08-19 journal: RESET WIPE at 16:23:38.699 (.cfg=deleted), "rebuilt a fresh topology" at .790, and a label cache at .890 with a distribution byte-identical to the pre-reset 2.00/2.00 one (Buy 3221 / Sell 3303 / Neutral 4837, mean lifespan 6.4 bars) - 100 ms later, with no DeriveBarrierGeometry between them. The member that actually blocked it is m_geometryDerivePasses. LoadAndCompareTopologyConfiguration pins it to BARRIER_DERIVE_MAX_PASSES to block the fixed-point iteration, which is correct for a LOAD - an existing model must never re-derive or the target moves under fitted weights - and exactly wrong for a RESET, which is the act of declaring there are no fitted weights left to protect. State that is correctly sticky for one lifecycle event, silently inherited by another; the same shape as the .cfg pin sitting beside it. ResetWeights now returns the whole derivation to its constructor state: the three latches, the pair, the horizon and its flags, the swing/lifespan measurements, the scan's own outputs, and m_spreadAtr - that last one matters because the cost filter is deliberately inert on pass 1 (m_spreadAtr <= 0.0) and an inherited spread makes a reset model walk a different ladder than a genuinely new one. m_sl_mode/m_tp_mode go back to the SL_Mode/TP_Mode inputs, since the adopt path overwrites them in place with no copy of what was configured. NOT COMPILED - user compiles in MetaEditor.
2026-08-19 16:31:18 -04:00
m_spreadAtr = 0.0;
m_barrierScanSlMult = 0.0;
m_barrierScanTpMult = 0.0;
m_barrierScanLiveLabels = false;
m_barrierScanTimeouts = 0;
//--- LoadAndCompare overwrites these in place when it adopts a trained pair, keeping no copy of
//--- what the user configured; leaving them adopted seeds the fresh derivation from the old pair.
fix(geometry): a weights reset could never change the barrier - the pair laundered itself through the wipe Reported as "it still seems leaned towards 2x atr" after a full Delete && Reset Weights on SP500 H4. It was not the .cfg pin, and it was not the new ratio floor failing to take: the geometry is held in members ResetWeights never cleared, so a reset wiped the .cfg, built a fresh net, restarted at era 0 - and then relabelled under the PREVIOUS model's pair, before SaveTopologyConfiguration wrote that stale pair back into the brand new .cfg. Reset, re-derive, re-pin, with the middle step missing. No number of resets could ever have moved it. Evidence in the 2026-08-19 journal: RESET WIPE at 16:23:38.699 (.cfg=deleted), "rebuilt a fresh topology" at .790, and a label cache at .890 with a distribution byte-identical to the pre-reset 2.00/2.00 one (Buy 3221 / Sell 3303 / Neutral 4837, mean lifespan 6.4 bars) - 100 ms later, with no DeriveBarrierGeometry between them. The member that actually blocked it is m_geometryDerivePasses. LoadAndCompareTopologyConfiguration pins it to BARRIER_DERIVE_MAX_PASSES to block the fixed-point iteration, which is correct for a LOAD - an existing model must never re-derive or the target moves under fitted weights - and exactly wrong for a RESET, which is the act of declaring there are no fitted weights left to protect. State that is correctly sticky for one lifecycle event, silently inherited by another; the same shape as the .cfg pin sitting beside it. ResetWeights now returns the whole derivation to its constructor state: the three latches, the pair, the horizon and its flags, the swing/lifespan measurements, the scan's own outputs, and m_spreadAtr - that last one matters because the cost filter is deliberately inert on pass 1 (m_spreadAtr <= 0.0) and an inherited spread makes a reset model walk a different ladder than a genuinely new one. m_sl_mode/m_tp_mode go back to the SL_Mode/TP_Mode inputs, since the adopt path overwrites them in place with no copy of what was configured. NOT COMPILED - user compiles in MetaEditor.
2026-08-19 16:31:18 -04:00
m_sl_mode = SL_Mode;
m_tp_mode = TP_Mode;
if(m_simOosRunActive)
{
delete m_simOosNet;
m_simOosNet = NULL;
m_simOosRunActive = false;
}
//--- Salted with this model's id so two members reset in the same millisecond cannot collide,
//--- and re-seeded so weight init is not dominated by the tuner's last candidate evaluation.
feat(rng): ALGLIB's L'Ecuyer generator replaces MathRand, and a seed collision goes with it MQL5's MathRand() is the 15-bit MSVC LCG - 32768 distinct values and the lattice structure that shape of generator has. Two places here actually lean on randomness and both were hurt by it: WEIGHT INIT. Six He/LeCun-uniform sites drew ((MathRand()+1)/32768.0 - 0.5) * 2 * scale, so a first dense layer of ~250k weights had only 32768 possible values and thousands of connections started byte-identical. Breaking that symmetry is the whole job of random init. SHUFFLING. ShuffleRandomIndex() already had to splice TWO MathRand() draws to reach 30 bits, and its own comment documented the residual modulo bias it still carried. HQRndUniformI() is rejection-sampled and exactly uniform, so the splice and the bias note both go. CHighQualityRand is L'Ecuyer's combined multiplicative congruential generator - two differenced streams, 31-bit output, period ~2.3e18 - and it ships with the terminal. AND A BUG THE MIGRATION EXPOSED. The three MathSrand(GetTickCount()) calls sit immediately before "build a fresh topology", once per model. GetTickCount() steps in ~15.6 ms on Windows and an ensemble builds every member inside one OnInit, so members could be handed the SAME seed and draw the SAME weights wherever their shapes coincide - and members that start identical are not an ensemble. WarriorRandSeed() takes a salt (the model id) plus a never-reset call counter, so a collision is impossible rather than merely unlikely, while the tick keeps the run itself genuinely unrepeatable the way those call sites asked for. Seeds are masked positive rather than trusted: HQRndSeed computes s % (M-1) + 1 and MQL5's % keeps the sign, so a negative seed leaves the generator in a state its own assertions reject. GetTickCount() is a uint and goes negative as an int after ~24 days of uptime - a fault that would surface as "training is broken" on a long-running terminal and nowhere else. The indicator tuner's 52 draws move across too: its random search is where sample quality earns its keep. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 00:29:12 -04:00
WarriorRandSeed(ID);
bool rebuilt = BuildFreshTopology();
if(!rebuilt)
Print(ID + ": ERROR - failed to rebuild fresh topology after weights reset");
else
{
//--- isInitialized=FALSE, never m_isInitialized: every other writer runs before init sets it
//--- true, so passing true here makes this the only .cfg on disk that fails its own compare
//--- on the next attach. Runtime lifecycle state must never gate reuse.
feat(nn): derive dense depth, train on all history, pin the shape in .cfg Completes the derived-topology work. Three inputs removed. AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five entries instead of eight. Depth is now derived from the two endpoints the taper already has to connect (derived first-layer width, output-tied final width) at a 2x per-layer compression target, clamped [2..5]. Asking a user to pick a layer count while the code derives the widths those layers taper between was asking for half a decision: at 64 units tapering to 12, four layers compress by 1.4x per step and five by 1.3x, so the extra depth bought no abstraction. On the shipping H1/10y default the derivation lands on 3 layers - the depth that actually won Run 2. StudyPeriods removed. There is no case for training on less data than the broker provides at a ~6% directional base rate; the honest generalization read comes from the OOS holdout, not from withholding history. Training now starts at the earliest available bar, floored by MinTrainYear, which answers a different question (excluding dubious pre-history) and stays. That required closing the hazard the old code documented: the capacity budget now MEASURES the symbol's real bar count, and a topology derived from a measurement would widen as history downloads. Both ends are now pinned. Every derived value left the weights-filename fingerprint - keying a filename on a measured quantity means the EA looks for a file that does not exist, starts from era 0 and orphans a trained model, silently, because a missing cache is the normal first-run state. The shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the four derived fields rather than diffing them; a mismatch there would discard a fully-trained model over nothing the user did. Two fields appended to the .cfg for the conv/LSTM stages, length-guarded on read because FileReadInteger past EOF returns 0 with no error. ForceHiddenLayers, a compile-time constant like DebuggingMode, pins depth for diagnostic comparisons. It joins the fingerprint only when non-zero, so forced depths get their own files - sequential comparisons only, not simultaneous from one .ex5. Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64, 3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from ~58k to ~28k weights. Both builds compile 0 errors, 0 warnings. Re-keys existing models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
SaveTopologyConfiguration(m_activeFileName, m_initialNeuronsCount, m_hiddenLayersCount, m_neuronsReduction, m_minNeuronsCount, m_optimizationAlgo, m_historyBars, m_outputNeuronsCount, m_neuronsCount, LEGACY_STUDY_PERIOD_SLOT, m_minTrainYear, false, LEGACY_CONVERGE_WR_SLOT, m_fractalPeriods, m_convFilterCount, m_lstmHiddenSize, m_activeFileCommon);
Print(ID + ": weights reset - training will restart from era 0 (current config only: " + m_activeFileName + ")");
}
m_trainingStopRequested = stopped;
if(!stopped && !bEventStudy)
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy Four user-reported/requested items, one root cause chain: 1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event id 1 and handled id 1001, and CExpertCustom broadcasts every chart event to every filter - so each posted event ran a train chunk in ALL N members (N*N chunks per round) and the chart thread never idled long enough to deliver clicks/drags. profiling.csv: 99.45% of time in OnChartEventHandler. Fix: per-instance study-event ids (STUDY_EVENT_ID_BASE + construction order, offset above the Controls library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel double-clicks fired training chunks). ArmStudyEvent() is the single post site; lost-event watchdog replaces the accidental sibling-clears-my-flag rescue. 2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over identical features/labels, and it ends in the full MI diagnostic suite, which the MI-share gate never intercepted on the sweep path - four members ran four identical ~36s sweep+report blocks. First member publishes outcome (g_ensembleChartTuneDone/Installed/Settings); the rest apply it and skip both. 3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy autosave in flight ate it, OnDeinit got ~430ms and died in the first member's arrow persist ("Abnormal termination" 432ms in). Fix: early visible-UI sweep (native prefix deletes for status/panel/dialog) right after ClearStatusLabel, and a fast path for still-training models - their arrows are re-rendered every era, so they get one bulk purge instead of scan+atomic-write in the death window. 4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era by era together; a member ahead of the slowest still-training member declines Train() calls and its chunk budget is donated (TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant). COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its adjusted per-bar decision (0.0 on abstain) to a shared row buffer; the last member to finish the era scores the averaged vote vs the mirrored Min_Vote_Open against the same target-before-stop outcomes members grade themselves on, publishing an "Ensemble vote" line on the aggregated panel. Member headlines now carry their lifetime win rate with break-even. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-16 19:06:04 -04:00
ArmStudyEvent(0, "Reset");
return rebuilt;
}
};
//+------------------------------------------------------------------+
//| IMPLEMENTATION. Method bodies, included after the declaration |
//| above and nowhere else. Order between them is irrelevant. |
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//+------------------------------------------------------------------+
#include "AIBase\Training.mqh"
#include "AIBase\Lifecycle.mqh"
#include "AIBase\Topology.mqh"
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
#include "AIBase\Labels.mqh"
#include "AIBase\OnlineLearning.mqh"
#include "AIBase\AutoTune.mqh"
#include "AIBase\Inference.mqh"
#include "AIBase\Persistence.mqh"
#include "AIBase\ChartUI.mqh"
#include "AIBase\Features.mqh"
feat: excursion-size head (Stage 1, measurement only) Direction is closed - normalised asymmetry fails on three instruments with a working positive control, and the classifier's own best-of-999 era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is a different question and RANGE clears at ~4x its null. Checked the denomination before building on that, since the source memo warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is predictable" is a claim about travel RELATIVE to current ATR, not a restatement of "ATR is autocorrelated". It is exactly the part a fixed multiple (stop 3.31*ATR, target 1.64*ATR) discards. A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches ladder rung k) upward and downward. Survival parameterisation rather than regressing the multiple, because it needs nothing new from CNet: sigmoid outputs and the per-neuron delta the `total != 3` branch already applies (a quantile head would need a linear activation and a pinball gradient in Network.mqh, Network.cl and the DirectML path, on a class four topologies share). Targets are free - m_ladderUpAt already records first-touch age per rung with 0 meaning never reached. Separate net, not extra outputs on the classifier: more outputs would change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push the count off 3 - the exact condition backProp uses to select the joint softmax gradient the 3-class head depends on. The classifier is bit-for-bit unaffected and this is removable without trace. STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the constant per-rung base rate - the baseline a fixed ATR multiple already assumes - with both predictors fitted IS and evaluated OOS, so neither gets a look at the test set. Positive skill justifies Stage 2 (drive SL/TP and sizing off ExcursionQuantile, which is defined and deliberately uncalled). Zero or negative means ATR already carries everything and Stage 2 must not be built. Trains only on primary occurrences: the replay queue oversamples for CLASS balance, and a direction-balanced sample is a biased SIZE sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
#include "AIBase\Excursion.mqh"
feat(gate): cross-instrument pooled certification The deploy bottleneck is CERTIFICATION, not training. A 4,738-bar OOS window at L=75.6 holds ~63 independent observations; certifying a 3pp edge at 2 sigma needs ~1,036. More bars of the same symbol barely help - they overlap. Other symbols do not. WHAT POOLS. Not win rates: symbols have different derived geometries, different break-evens and different drifts, so averaging raw rates across them is meaningless. What pools is each symbol's EXCESS OVER ITS OWN CHANCE RATE, combined by inverse-variance weighting (fixed-effects meta-analysis). Each symbol keeps its own model, geometry and chance rate; only the evidence is combined. THE CORRELATION PROBLEM, bracketed rather than assumed away. SP500 and NAS100 are ~0.9 correlated and pooling them as independent inflates the evidence. Nothing here can measure that without sharing return series, so instead of guessing a correction the gate reports both ends: SE_INDEP = sqrt(1/SUM(1/var_i)) all members independent SE_CORR = SUM(w_i * sqrt(var_i)) all members perfectly correlated The truth is always between. THE GATE USES SE_CORR, so a pass cannot be an artifact of correlated instruments - that bound already assumes the worst. The ratio is logged as the diversification credit the gate declines to claim, so the cost of that conservatism is visible instead of hidden. SCOPE, deliberately limited: the pooled result is REPORTED, never folded into tradeableOK. The local gate certifies the model that actually trades this symbol; the pool answers the different question of whether the strategy has an edge at all. Letting a cross-symbol result license a local deploy would ship a model that never cleared its own bar - so it cannot. Mechanics: one file per instrument (no concurrent-write path to get wrong), every FileOpen carrying FILE_SHARE_READ|FILE_SHARE_WRITE, records skipped rather than reinterpreted on a version mismatch, 12h staleness cutoff so a stopped chart cannot vote, and pooling refused below 3 instruments. Poolability requires matching timeframe and ratio; differing SYMBOL is the entire point. Publishing is unconditional - a pool that only hears from winners is a selection effect, not a meta-analysis. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:45:37 -04:00
#include "AIBase\PooledGate.mqh"
feat(baselines): Alglib forest + linear on the NN's own matrix Every direction verdict so far was measured through one architecture family, so "flat" has two readings that no topology tuning can separate: the net is the wrong learner, or the matrix carries no directional information. Two learners with completely different inductive biases - Alglib's random decision forest and an ordinary least-squares fit - now train on the SAME feature windows (BuildFeatureWindow, the net's own function, so there is no second feature implementation to drift), the SAME labels, the SAME IS/OOS split with both purges, and are scored through the SAME precision-against-always-one-direction comparison and the same Sidak family-wise arithmetic the deploy gate uses. If both also land at chance, the matrix is the limit. Deliberate choices, each of which could have made the comparison a different question wearing this one's name: - LRBuild, not LRBuildZ: the intercept absorbs the class imbalance, and a baseline handicapped by a forced zero intercept would flatter the net for the wrong reason. - Raw call counts in the SE, matching the live gate's known-permissive test rather than correcting it here - both sides must face the same bar. - No threshold sweep on the linear fit: a threshold fitted on the slice being scored is the calibration leak the purged band exists to avoid. - Uniform stride when a cap bites, not the newest N rows, so a score difference cannot be a regime difference. What was dropped is logged. Ships off (Run_Alglib_Baselines = false): it is a measurement, not a trading feature, nothing trades on the answer and no model is saved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-19 23:45:03 -04:00
#include "AIBase\Baselines.mqh"
//+------------------------------------------------------------------+