Warrior_EA/Expert/ExpertSignalAIBase.mqh

3292 lines
254 KiB
MQL5
Raw Permalink Normal View History

feat(ai): logit-adjusted loss, replacing oversampling and the post-hoc prior Menon et al. 2021 (ICLR), "Long-tail learning via logit adjustment": add tau*log(prior_c) to each class logit inside the training gradient. Softmax CE on adjusted logits is consistent for BALANCED error - the metric checkpoint selection already ranks on - so the loss and the deploy decision finally optimize the same thing. The engine already computed a true softmax + categorical-CE gradient and wrote it over the per-neuron sigmoid delta, so this is an offset added to three logits in the two places that gradient is built (backProp scalar path and backPropOCL). No backend, kernel or DLL change; the forward pass and every inference path are untouched, which is the point - the network learns to absorb the offset, so its raw argmax becomes the balanced-optimal decision with nothing applied at inference. Replaces rather than stacks. Minority replay is disabled while this is on, and the post-hoc inference prior is forced off. Stacking is not a theoretical worry: simulated on the measured 1118/1119/34298 distribution in the weak-signal regime, plain CE collapses to Neutral (33.4% balanced, Buy 0%), replay reaches 48.1%, logit adjustment 50.9% with better balance - and BOTH together score 45.4% with Neutral recall at 0%, worse than either alone. Buda et al. 2018 predicts exactly that. Motivation from the six-chart run: every topology took one direction to ~50% recall and abandoned the other, the direction chosen arbitrarily (the batch-norm control went Buy 1% / Sell 42%, the inverse of the other five). One era in 1,301 cleared the per-class recall floor. Fingerprinted conditionally, so the converged 60.7% models on disk keep their filenames and stay loadable as the fallback. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:05:14 -04:00
//+------------------------------------------------------------------+
//| Warrior_EA |
//| AnimateDread |
//| |
//+------------------------------------------------------------------+
#include "ExpertSignalCustom.mqh"
#include "..\AI\Network.mqh"
#include "..\Variables\IndicatorResources.mqh"
#include "..\Variables\IndicatorTuneRanges.mqh"
#include "..\System\StatusLabel.mqh"
#include "..\System\NewsRelevance.mqh"
fix(signals): revive a dead MA model, and demote Sanyaku from state to event Two defects surfaced by research/test_classic.py, both verified fixed by re-running the transcription against 178k bars of EURUSD H1. CSignalMA model 1 could never fire. For any recursive average - and MA_TYPE_EMA is the shipped default - MA(i) = a*Close(i) + (1-a)*MA(i+1), so DiffMA(i) = a * (Close(i) - MA(i+1)) DiffCloseMA(i) = (1-a) * (Close(i) - MA(i+1)) are positive multiples of one quantity and always share a sign. Model 1 asks for a close BELOW a RISING average, which is precisely the combination that identity forbids: 0.000% of bars, either direction, any symbol. The MQL5 standard library this was ported from defaults to MODE_SMA, where the two are merely correlated - the bug arrived with the EMA default, not with the port. Reading the slope one bar back (DiffMAPrev) breaks the tie for every MA type while keeping the model's stated meaning. Now fires on 7.92% of bars. CSignalIchimoku model 11 fired on 27% of bars at weight 100. Sanyaku is three standing STATES conjoined with no transition term, so it held across long stretches - and being last in the if-chain at the top weight, the module's highest-conviction reading was also its most common one, overwriting all eight event models below it on a quarter of all bars. The old comment rejected an event form because "demanding all three flip on the same bar would fire almost never" - true, but that is not the alternative. Kouten is the TURN: the ALIGNMENT transitions, and only one role need change for it to. Testing !Sanyaku(idx+1) fires once per aligned stretch. Now 2.17%, in line with Kumo breakout (2.4%) and the strong TK cross (1.1%). DataReady() extended one bar deeper to cover the lookback. Neither pattern showed edge before or after; this is about the models meaning what they say and the vote not being dominated by a constant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:14:34 -04:00
#include "..\System\CrossAsset.mqh"
#include "ADIndicatorTuner.mqh"
//--- Hard 0/1 one-hot targets for the 3-neuron classification head - matches the book's
//--- (nnbook.txt sec. 1.4 "Cross-entropy") cross-entropy formulation, which defines the reference
//--- distribution's occurring-event probability as exactly 1.0 (missing event exactly 0.0), no
//--- smoothing. This used to be softened to 0.9/0.1 (see git history) as a workaround for the OLD
//--- per-neuron independent-sigmoid backward gradient: since that gradient was each neuron's own
//--- raw (target-sigmoid_output) delta with nothing else driving it to zero, a literal 1.0/0.0
//--- target was only ever approached asymptotically, so weights kept growing (up to the MAX_WEIGHT
//--- clamp) chasing it forever. Now that CNet::backProp()/backPropOCL() (AI\Network.mqh) compute a
//--- joint softmax+categorical-cross-entropy gradient (softmax_i - target_i) across all 3 neurons
//--- instead, the winning class's SOFTMAX probability - not any one neuron's raw sigmoid value -
//--- is what needs to approach the target, and softmax can reach very close to 1.0 for the winning
//--- class from ordinary (non-saturated) logit separation, so the old runaway-weight failure mode
//--- is no longer expected to require smoothed targets to avoid. If it resurfaces in practice (the
//--- per-neuron SIGMOID forward pass can still individually saturate before softmax normalizes),
//--- that's the first thing to check before reintroducing smoothing.
// Soft labels keep the classification head from overfitting to hard one-hot targets while still
// preserving a clear target for the true class. The true class gets 0.9 and the others 0.05 each.
#define LABEL_SMOOTH_HIGH 0.9
#define LABEL_SMOOTH_LOW 0.05
//--- Namespace prefix for the directional signal arrows this class draws (DrawObject/DeleteObject). Two
//--- reasons it exists: (1) so PurgeChart() can delete ONLY our arrows and never the user's own manual
//--- chart drawings (a full ObjectsDeleteAll(0) on a client's chart is not acceptable for a commercial
//--- product), and (2) so arrows can survive an EA re-init instead of being wiped on every InitIndicators
//--- - the whole point of keeping them on screen (the panel has a hide toggle if the user wants them gone).
#define SIG_ARROW_PREFIX "WarSig_"
fix: purge every EA object namespace on init and after deinit teardown Leftover objects survived deinit because the cleanup list had drifted. PurgeChart()'s own comment said it removed "our namespaced signal arrows plus the status-label objects" while the code removed arrows ONLY, and the panel prefix was swept at OnInit and nowhere else - so an ordinary deinit left the status line, and any panel straggler, on the chart. Three scattered call sites and a comment cannot be kept in step. There is now ONE list - WarriorChartPrefixes() - covering arrows, status label and panel, and one sweep, WarriorPurgeChartObjects(), used by every path. Add a prefix there when a new object family appears and every cleanup picks it up. Two call sites added: OnInit, before ANYTHING is drawn (including the status label it would otherwise delete). Chart objects live in the chart PROFILE, not in the EA, so they outlive the process: a deinit force-terminated at MetaTrader's ~4,500 ms budget, a crash, a terminal kill, or an .ex5 replaced while attached all strand objects no later deinit will ever own - and deleting the EA's files does not remove them, which is why they read as corruption. Arrows are included: LoadChartSignals restores them from their sidecar moments later and already opens with its own arrow sweep, so this only removes orphans the sidecar does not account for - the ones SaveChartSignals would otherwise ADOPT, since it rebuilds that sidecar by scanning the chart. OnDeinit, after ExtPanel.Destroy. Destroy walks an unbounded control tree and ClearStatusLabel clears text rather than guaranteeing object removal; either can leave a straggler and nothing looked afterwards. Bounded work - three prefix deletes and one object-list scan - so it respects the ordering rule that keeps the cheap visible cleanup ahead of the heavy save. Arrows excluded: ShutdownChartCleanup already persisted and removed them and re-deleting would race that write. The two are complementary: the deinit sweep closes the ordinary case, the OnInit purge closes the case where MetaTrader never let us finish. Only the second can help after a starved shutdown. Both sweeps rescan by name across EVERY object type and delete what the bulk call missed. ObjectsDeleteAll's return has already been observed disagreeing with a by-name scan of the same chart microseconds apart, and object commands are queued on the chart rather than applied inline, so a returned count is not evidence the objects are gone. Panel create site now uses WARRIOR_PANEL_PREFIX instead of a literal, so the name cannot drift away from the list that cleans it up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:19:26 -04:00
//--- Control-panel object namespace. CAppDialog names every control it owns from the dialog name it is
//--- created with, so one prefix covers the whole tree. Declared HERE rather than left as a string
//--- literal at the ExtPanel.Create call site so it can appear in the sweep list below - a prefix that
//--- lives only at its creation site is a prefix nothing cleans up.
#define WARRIOR_PANEL_PREFIX "WarriorCP"
//+------------------------------------------------------------------+
//| EVERY chart-object namespace this EA creates, in ONE list. |
//| |
//| This exists because the list drifted. PurgeChart()'s own comment |
//| said it removed "our namespaced signal arrows plus the status- |
//| label objects" while the code deleted arrows only, and the panel |
//| prefix was swept at OnInit and nowhere else - so a deinit left |
//| the status line and any panel straggler on the chart, which is |
//| exactly the reported symptom. A comment cannot be kept in step |
//| with three scattered call sites; one array can. |
//| |
//| Add a prefix here the moment a new object family is introduced. |
//| Deleting by prefix and never by ObjectsDeleteAll(chart) is |
//| deliberate: a blanket wipe also removes the user's own drawings |
//| and other indicators' objects, which is not acceptable on a |
//| client's chart. |
//+------------------------------------------------------------------+
int WarriorChartPrefixes(string &out[])
{
ArrayResize(out, 3);
out[0] = SIG_ARROW_PREFIX; // directional signal arrows (ChartUI DrawObject)
out[1] = STATUS_LABEL_PREFIX; // status line background + text (System\StatusLabel.mqh)
out[2] = WARRIOR_PANEL_PREFIX; // control panel and its whole control tree
return 3;
}
//+------------------------------------------------------------------+
//| Delete every object in those namespaces from a chart, and verify. |
//| |
//| skipArrows leaves the signal arrows alone, for the one caller |
//| that must: a re-init restores arrows from their sidecar and |
//| wiping them here would make them flicker off and back on. |
//| |
//| The rescan is not paranoia. ObjectsDeleteAll's return value was |
//| already observed disagreeing with a by-name scan of the same |
//| chart microseconds apart, and object commands are QUEUED on the |
//| chart rather than applied inline - so "the bulk call returned a |
//| number" is not evidence the objects are gone. Names are collected |
//| before any deletion because deleting while enumerating by index |
//| renumbers the list being walked. |
//+------------------------------------------------------------------+
int WarriorPurgeChartObjects(long chartID, bool skipArrows, int &leftoverCount)
{
string prefixes[];
int n = WarriorChartPrefixes(prefixes);
int removed = 0;
leftoverCount = 0;
for(int p = 0; p < n; p++)
{
if(skipArrows && prefixes[p] == SIG_ARROW_PREFIX)
continue;
int r = ObjectsDeleteAll(chartID, prefixes[p]);
if(r > 0)
removed += r;
}
//--- Typed-blind rescan across EVERY object type: an earlier version filtered on OBJ_ARROW and was
//--- therefore blind in the same way the bulk delete was, which is how two scans of one chart
//--- disagreed for three sessions.
int total = ObjectsTotal(chartID, -1, -1);
string leftovers[];
int found = 0;
if(total > 0)
{
ArrayResize(leftovers, total);
for(int i = 0; i < total; i++)
{
string nm = ObjectName(chartID, i, -1, -1);
for(int p = 0; p < n; p++)
{
if(skipArrows && prefixes[p] == SIG_ARROW_PREFIX)
continue;
if(StringFind(nm, prefixes[p]) == 0)
{
leftovers[found++] = nm;
break;
}
}
}
}
for(int i = 0; i < found; i++)
ObjectDelete(chartID, leftovers[i]);
leftoverCount = found;
return removed + found;
}
//--- Upper bound on arrows restored from a .arrows file (see LoadChartSignals). Purely a guard against a
//--- corrupt header declaring a garbage count - a long converged run legitimately accumulates a few
//--- thousand, so this sits well above that. Restoring is chunked across timer calls regardless, so a
//--- large-but-valid count costs progressive fill-in, never a frozen OnInit.
#define MAX_RESTORED_ARROWS 50000
//--- How many of the MOST RECENT arrows are kept on the chart and in the .arrows sidecar. Arrows would
//--- otherwise accumulate for the life of the model (a converged SP500 H1 run had reached 2896), which
//--- clutters the chart, slows every save/restore, and preserves history nobody scrolls back to. Both
//--- SaveChartSignals (which also DELETES the pruned objects from the chart) and LoadChartSignals select
//--- by TIME, not by scan order - ObjectsTotal() order is arbitrary, so "the last N scanned" would keep a
//--- random subset rather than the newest.
#define MAX_PERSISTED_ARROWS 1000
//--- Manual "rescan" window (see RescanChartSignals): how many of the MOST RECENT bars get re-inferred
//--- from the currently deployed weights when the operator asks for a fresh signal set. Bounded well
//--- below a full StudyPeriods re-render (which can be years of bars and would freeze the one MQL5 chart
//--- thread, same doctrine as MAX_RESTORED_ARROWS/ARROW_RESTORE_BUDGET_MS) - a rescan is meant to replace
//--- stale old arrows with what the model calls on RECENT history, not reproduce the whole training run.
#define SIGNAL_RESCAN_LOOKBACK_BARS 5000
//--- Wall-clock budget per chunk of the deferred arrow restore. Same doctrine as TRAIN_TIME_BUDGET_MS:
//--- MQL5 gives a chart ONE thread, so "async" here means small time-boxed slices between which the
//--- terminal can service the panel, the chart and the journal - never a single long blocking pass.
//--- 50ms sits between the training chunk (120ms, already tolerated) and the 500ms timer period, so the
//--- restore completes in a handful of slices while leaving ~90% of each timer window free for the UI.
#define ARROW_RESTORE_BUDGET_MS 50
//--- Max allowed |Δ| between the compute backend's and the pure-MQL5 path's outputs for a model to be
//--- marked MQL5-inference-safe (see ValidateCpuInference). The outputs are bounded sigmoid values;
//--- backend-vs-double summation-order noise is ~1e-6 on the CPU-DLL path (double) and stays well under
//--- this even on a float32 OpenCL backend, while a genuine math/port bug shows up as >0.01. Fails safe.
#define CPU_INFERENCE_MAX_DIFF 1.0e-3
extern bool g_signalsVisible;
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- FILTER-BASED auto-tuner constants (see TuneIndicatorsByFilter). The GA_* / TUNE_POP_* knobs that
//--- lived here are gone with the genetic search they configured.
//--- MI_BINS: equal-frequency bins the feature column is discretised into before the joint histogram.
//--- Mutual information is biased upward as bins increase (each bin holds fewer samples, so noise looks
//--- like structure); 8 bins against MI_SAMPLE_BARS samples keeps ~250 samples per bin per class, which
//--- is comfortably in the regime where that bias is small and equal across candidates - and equal is
//--- what matters, since this score is only ever used to RANK.
#define MI_BINS 8
//--- Bars sampled (evenly spaced across the in-sample window) per candidate evaluation. The whole cost of
//--- tuning is candidates x this x features, so it is the one number that trades accuracy for time.
#define MI_SAMPLE_BARS 2000
#define MI_MIN_SAMPLES 200
//--- Eras the MI diagnostics may wait for the cross-asset panel before reporting without it. Three is
//--- enough for the terminal to finish synchronising auxiliary symbols on a live chart, and short
//--- enough that a tester run - where an un-downloaded reference symbol is permanently absent - is not
//--- left without diagnostics at all.
#define MI_REPORT_MAX_DEFERRALS 3
//--- Largest |k| the label-alignment scan uses. Every MI sample is padded by MiShiftPad() - which is at
//--- least this - at BOTH ends REGARDLESS of the offset being requested, so that every build enumerates
//--- the IDENTICAL bar set with the IDENTICAL stride. That is what makes two builds comparable row by
//--- row. Padding by |offset| instead (as this did until 2026-08-02) shifted the offset build's starting
//--- bar, so the positive control paired rows that were offset+pad apart rather than offset apart: it
//--- reported 0.00307 nats for a pair it called "24 bars apart", which is the value for 48 bars, failed
//--- its own 5x gate, and printed "every mutual-information figure above is void" over perfectly sound
//--- measurements. Verified against an independent computation in research/test_mi_control.py.
#define MI_ALIGN_MAX_SHIFT 5
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- Coordinate-descent passes. The second pass lets a parameter re-optimise against what the others
//--- moved to; the loop breaks early as soon as a pass changes nothing, so this is a ceiling, not a cost.
#define MI_TUNE_PASSES 2
diag(autotune): five permutations was still a coin flip - use a real test The 5-draw z-score shipped an hour ago disproved itself on its first run. All four charts scored the IDENTICAL 0.00401 nats on identical features and identical labels - and reported z of +1.3, +2.0, +4.0 and +4.7. Two "AT THE NOISE FLOOR", two "a real association", same data. The entire swing came from estimating the null's spread from five draws, where the standard deviation of the standard-deviation estimate is ~35%: the denominator was noisier than the effect it was judging. Replaced with an empirical permutation test. 200 draws, p counted by rank with the +1/(B+1) correction (Phipson & Smyth 2010) so p is never reported as exactly zero - no normality assumption and no spread to estimate. The strongest single column is tested against the null distribution OF THE MAXIMUM, which corrects for scoring 26 features at once by construction and is far less conservative than Bonferroni. Affordable because BuildMiSample is now split out of ScoreCurrentParamsByMI and runs ONCE for the whole test - every draw reuses that sample and costs a relabel plus 26 histogram passes, not 2000 feature extractions. The coordinate sweep still calls the combined form, which is correct there: each candidate changes the indicator settings, so its features really do have to be re-extracted. The verdict line keeps both questions apart and prints both answers: the p-value for "is it real", the excess as a percentage of H(Y) for "is it big enough to trade". At n=2000 those can disagree, and collapsing them into one word is how a worthless effect gets called a discovery. Compiles 0 errors / 0 warnings. Build tag permtest-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:45:46 -04:00
//--- MI_NOISE_PERMUTATIONS: draws from the null distribution used to test the observed score. Sets the
//--- resolution of the empirical p-value, which can never go below 1/(B+1) - so 200 draws can report
//--- "p<=0.005" and no finer, which is ample for a yes/no on whether a feature set carries signal.
//--- Cheap because BuildMiSample runs ONCE and every draw reuses it (see ScoreMiSample); the cost is a
//--- relabel and 26 histogram passes, not 2000 feature extractions.
//--- Two earlier values were wrong and both are instructive. ONE shuffle answers "is this above a coin
//--- flip's worth of noise" rather than "is this above noise". FIVE looked principled but was not: on
//--- 2026-08-01 all four charts scored the identical 0.00401 nats on identical features and identical
//--- labels, and reported z of +1.3, +2.0, +4.0 and +4.7 - two "noise floor", two "real". With five
//--- draws the standard deviation of the standard-deviation estimate is ~35%, so the denominator of that
//--- z was noisier than the effect it was judging. Counting ranks avoids estimating a spread at all.
#define MI_NOISE_PERMUTATIONS 200
diag: MI feature-lag profile - close the blind spot in every MI verdict so far BuildMiSample samples features from ONE bar. So every "MI is at the noise floor" result this codebase has produced - including yesterday's p=0.18 on SP500 H1 - described the ENTRY BAR's 31 features only, while the network is fed 20 bars of them. If information lived at lag 7 and not lag 0, the report would have said "no signal" while the model could still learn. The diagnostic we have been making decisions on had a blind spot exactly the width of the input vector. Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as the existing labelBarOffset and is not interchangeable with it. Shifting the LABEL changes which trade is predicted, so at any non-zero offset the features sit inside the labelled window and the score is lookahead - that is precisely what the alignment scan measures and correctly reports (4.7x more knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label pinned to the entry bar, so every row stays causal. ReportFeatureLagProfile() then scores k = 0..historyBars against the same block-permutation null and reports the deepest lag that clears it - the lookback the data supports, versus the 20 that was picked by hand and never measured. The null is redrawn PER LAG: finite-sample MI bias moves with the realised class counts and bin occupancy, and different rows survive the validity checks at each lag, so one shared floor would be right for lag 0 and wrong everywhere else. Draw count is reduced accordingly (40, not 200) since cost is draws x historyBars; this figure decides a lookback, never a trade. MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that makes two builds comparable row by row. Read-only - no input, topology or label change, so no retrain. Both builds 0/0. Build tag lag-profile-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 13:51:39 -04:00
//--- Lag-profile draws, deliberately far below MI_NOISE_PERMUTATIONS: the profile redraws its null at
//--- EVERY lag (see ReportFeatureLagProfile for why a shared floor would be wrong), so the cost is
//--- draws x historyBars, not draws. 40 resolves a p of 0.05 to within about one draw, which is all a
//--- lookback decision needs - this figure never gates a trade.
#define MI_LAG_PERMUTATIONS 40
fix: correct the lag profile across lags too - it contradicted itself 3271f1e tested each of ~21 lags against its OWN null at alpha 0.05 and starred whatever cleared. That is about one false positive per run before any signal exists, and because neighbouring lags share nearly their entire feature window the false positives arrive in CLUSTERS that read like a hump. It did exactly that on SP500 H1, twice in one afternoon on identical data: 13:55 nothing clears at any lag headline MI p=0.4478 16:22 k6/k10/k12/k16 starred, headline MI p=0.8756, observed "information survives to lag 16" BELOW its own null mean Same 31 features, same 2009 samples, same 287 blocks, cross-asset absent in both - so this was not two different measurements. Non-replication on identical data is the signature of an uncorrected multiple comparison, and acting on the second run would have pinned the lookback to 17 off noise. Galling detail: 04ee2e1 had just added exactly this correction to the barrier-geometry scan one function below. The rigorous bar went on the report with 6 candidates and the naive one stayed on the report with 21. So the lag profile now uses the same construction as the geometry winner test: one draw from every lag, keep the largest, repeat; a lag clears only by beating that distribution. Draws centred leave-one-out to match how the observed excess is centred. Independence across lags overstates the spread of the maximum (neighbours share their window), so it errs toward rejecting. Also: the positive branch now says to re-run before acting, because one run of this report has demonstrably not been a result; and MI_LAG_MAX_PROFILE caps the retained-draw matrix rather than trusting a derived m_historyBars. Read-only diagnostic. No input, topology or label change: no retrain, and a training run already in flight stays valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 16:28:57 -04:00
//--- Per-lag significance, applied against the null of the MAXIMUM over lags rather than against each
//--- lag's own null. The first version did the latter and it was wrong: ~21 lags at 0.05 stars one lag
//--- per run before any signal exists, and on SP500 H1 that produced two opposite verdicts on identical
//--- data hours apart. The alpha stays 0.05; what changed is the null it is measured against.
diag: MI feature-lag profile - close the blind spot in every MI verdict so far BuildMiSample samples features from ONE bar. So every "MI is at the noise floor" result this codebase has produced - including yesterday's p=0.18 on SP500 H1 - described the ENTRY BAR's 31 features only, while the network is fed 20 bars of them. If information lived at lag 7 and not lag 0, the report would have said "no signal" while the model could still learn. The diagnostic we have been making decisions on had a blind spot exactly the width of the input vector. Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as the existing labelBarOffset and is not interchangeable with it. Shifting the LABEL changes which trade is predicted, so at any non-zero offset the features sit inside the labelled window and the score is lookahead - that is precisely what the alignment scan measures and correctly reports (4.7x more knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label pinned to the entry bar, so every row stays causal. ReportFeatureLagProfile() then scores k = 0..historyBars against the same block-permutation null and reports the deepest lag that clears it - the lookback the data supports, versus the 20 that was picked by hand and never measured. The null is redrawn PER LAG: finite-sample MI bias moves with the realised class counts and bin occupancy, and different rows survive the validity checks at each lag, so one shared floor would be right for lag 0 and wrong everywhere else. Draw count is reduced accordingly (40, not 200) since cost is draws x historyBars; this figure decides a lookback, never a trade. MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that makes two builds comparable row by row. Read-only - no input, topology or label change, so no retrain. Both builds 0/0. Build tag lag-profile-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 13:51:39 -04:00
#define MI_LAG_ALPHA 0.05
diag: is "optimal SL/TP" learnable? Score the features against excursions Proposed direction: train the net to predict entry/SL/TP that maximise return and minimise drawdown, rather than to classify direction. Before rebuilding a head, measure whether the target is learnable at all. That question splits into two that behave nothing alike: HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility clustering is about the most robust regularity in markets. WHICH WAY it goes first (the asymmetry) - direction, which is what every noise-floor verdict in this project has been about. Expectancy comes ONLY from the second. The first buys position sizing and drawdown control - worth having under prop-firm limits, but not an edge: exit management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with expectancy FLAT. Crucially this is NOT already answered. Every MI figure here scored the triple-barrier label, i.e. one specific question at one fixed geometry. A noise-floor result there says nothing about whether excursion MAGNITUDE is learnable - different target, different answer. Four targets, and the verdict is the CONTRAST, printed explicitly because the dangerous misreading of "UP clears" is "we can predict profitable trades": RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that SHOULD clear. Every prior verdict here lacked a control expected to pass; a range target at the floor indicts the measurement, not the market. UP / DOWN - MFE / MAE. ASYMMETRY - up-dn, the only one that can pay. Collected inside the walk the label already does (one max, one min per bar). The early-out when both barriers resolved is GONE: it would have truncated the excursions at whichever bar tripped the last barrier, making the measurement a function of the CURRENT SL/TP - the circularity this is trying to escape. The loop was already bounded by the horizon, so only the average cost moves. Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is fat-tailed and fixed-width bins would put nearly every row in bin 0; it also pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and to the barrier label's ~1.02 instead of confounded by class balance. Two bugs fixed in this code before it ever ran, both of which would have produced a plausible quiet wrong answer rather than an error: - TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the accumulators were reset, so one bar's excursions would be cached under another bar's index. Cleared at the top now, ahead of every return. - An unresolvable bar is still flagged as labelled but carries excursions of exactly 0. Under equal-frequency binning a block of identical zeros drags the lowest cut onto zero and a third of the sample lands in one uninformative bin - a depressed score that reads as "not predictable", a false negative in the direction that would wrongly kill the idea. Rows where both excursions are zero are dropped; price cannot travel zero both ways over a whole horizon. Read-only diagnostic. No topology or label change: no retrain of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 10:22:41 -04:00
//--- WHICH TARGET BuildMiSample() scores the features against. The barrier class is the shipped training
//--- target; the excursion targets exist to answer a question the barrier label cannot, and that every MI
//--- verdict in this project so far has silently conflated.
//---
//--- "Optimal SL/TP" decomposes into two predictions that behave nothing alike:
//--- HOW FAR price travels (the excursions) - essentially a volatility question, and volatility
//--- clustering is about the most robust regularity there is, so expect this to be predictable.
//--- WHICH barrier is reached first (the asymmetry) - direction, which is what the barrier label
//--- measures and what has come back at the noise floor every time.
//--- Expectancy comes ONLY from the second. The first buys position sizing and drawdown control, which
//--- is worth having under prop-firm limits but is not an edge - exit management tested against RANDOM
//--- entries moved the payoff ratio 0.92 -> 5.72 with expectancy FLAT.
//--- Measuring them separately is the point: if size clears and asymmetry does not, the deliverable is
//--- risk control and we should stop looking for edge in the exit.
fix: normalise the asymmetry target - the raw one is confounded by volatility Three symbols ran the excursion test. RANGE/UP/DOWN cleared on all three; raw ASYMMETRY cleared on EURUSD and USDCAD at p=0.0050 and not on SP500 (p=0.1045). That looked like the first directional signal this project has found. It probably is not, and the test as built could not tell. (up-dn) IS NOT SCALE-FREE. If sigma is predictable - and RANGE clears at ~4x its null on every instrument - and the directional part is symmetric noise eps, then up-dn ~ sigma*eps, so a large sigma pushes the value into BOTH outer terciles. A pure volatility predictor scores positive MI against a 3-bin (up-dn) while carrying no directional information at all. Crucially that confound REPLICATES, so reproducing on two instruments is not evidence against it - and the effect sizes fit it: asymmetry runs 1.3-1.6x its null where RANGE runs ~4x, and carries ~0.1% of the target's entropy against RANGE's ~0.9%. That is the shape of a leaked fraction of the volatility signal, not an independent one. So add (up-dn)/(up+dn): bounded in [-1,+1], volatility divided out, and the only target a directional claim may rest on. The verdict now separates the cases and NAMES the confound when raw clears while normalised does not, instead of reporting the raw line as a finding. Two bugs of mine in the same block, both caught by output rather than review: - The derived-geometry line had a MISORDERED argument list: it printed "stop 25.00*ATR (q3 of adverse travel)" - the quantile percentage as the multiple and the multiple as the quantile. Real values were 2.61 stop / 8.03 target. A 25*ATR stop is absurd on its face, which is why it was seen. - THE STOP QUANTILE WAS BACKWARDS, and this one changes labels. It was 0.25 "so ordinary noise does not reach it", but q25 means 75% of bars EXCEED the stop - hit three times in four. The printed reachability said exactly that ("stop on 75.0% of bars"). Now 0.75. A quantile is a threshold, not a rate. This is the entire reason reachability is measured and printed rather than assumed. Also raises BARRIER_DERIVE_MAX_PASSES 3 -> 5: SP500 did not settle in 3 (stop still moving ~14% per pass) while EURUSD and USDCAD converged on pass 2. And bounds both quantile indices with MathMin(..., n-1) so q=1.0 cannot run off the end of the sorted array. The geometry from the previous run is NOT usable and the asymmetry result is unresolved, not established. Both are decided by the next run. FORCES A FULL RETRAIN (the stop quantile changes every label). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 13:04:13 -04:00
//--- BARRIER DERIVATION. The stop sits at a HIGH quantile of adverse travel, so only the minority of bars
//--- whose adverse excursion exceeds it ever reach it; the target at the MEDIAN of favourable travel, so
//--- it is reached about half the time inside the horizon by construction. Neither number creates
//--- expectancy (chance precision equals break-even at every geometry) - they make the target reachable
//--- and the stop survivable, which the enum grid {2,3} x {2,3,4,6,8,10} could only do by luck.
//---
//--- THE STOP QUANTILE WAS 0.25 AND THAT WAS BACKWARDS. q25 means 75% of bars exceed the stop, i.e. it is
//--- hit three times in four - the opposite of "ordinary noise does not reach it". Caught by the measured
//--- reachability line the derivation prints ("stop on 75.0% of bars"), which is the entire reason that
//--- figure is reported instead of assumed. A quantile is a threshold, not a rate: to be rarely reached a
//--- stop must sit ABOVE most of the distribution.
#define BARRIER_SL_QUANTILE 0.75
feat: derive the ATR multiples from measured excursions - no hardcoded geometry The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in 3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of eleven guesses is not deriving anything. WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It ranks pairings by how predictable their OUTCOME is - a question about direction. The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x. Hence the scan failing its own gate on every run, and its "winner" wandering 2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is strongly measurable, so derive the geometry from that instead. stop = q25 of measured ADVERSE travel (ordinary noise does not reach it) target = q50 of measured FAVOURABLE travel (reached ~half the time, by construction, inside the horizon) Continuous, in ATR units, superseding the enum multiples. Reachability ("target on X% of bars, stop on Y%") and the implied break-even are printed so the choice is auditable rather than trusted. FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the horizon with the target (first-passage time grows with the band) and the excursions are measured OVER the horizon, so target -> horizon -> excursions -> target is a real loop - deriving once sizes the target from travel measured under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at 3 passes, and says so if it does not settle. Does NOT create expectancy, and the log says as much: chance precision equals break-even at every geometry (m/(m+k) on both sides). It buys a target the market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio forces a target the market rarely reaches, it WARNS rather than overriding - the ratio is the user's risk policy, so the honest move is to state its cost. That is the collision that once rejected 100% of setups. Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg files written earlier today still load (their length guard finds no doubles) and a model that carries them was trained on them and never re-derives. Also fixes a message from e5ceed6 that claimed "this model resumed from disk" unconditionally - it printed above a "seeding era 0" line on a brand-new model, because the branch fires whenever the cache is not built, which is equally true before a fresh model's first prebuild. A diagnostic that misreports its own trigger is worse than one that says nothing: it gets quoted back as evidence. FORCES A FULL RETRAIN (labels change). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 12:06:25 -04:00
#define BARRIER_TP_QUANTILE 0.50
//--- Below this many resolved excursions the quantiles are too noisy to key a training target on, and the
//--- configured multiples stand.
#define BARRIER_DERIVE_MIN_SAMPLES 500
feat: first-passage ladder + expectancy scan - price every geometry, not just the chosen one Corrects the premise of the previous plan. Break-even is NOT a ceiling. If the model shifts the win probability on the bars it selects from p0 = m/(m+k) to p0 + d, then EV = (p0+d)*k - (1-p0-d)*m = d*(k+m) because p0*k - (1-p0)*m is zero by construction. The stop:target RATIO is expectancy-neutral - a punishing break-even is exactly repaid by the payoff - and only the real edge d and the TOTAL WIDTH (k+m) move EV. Width matters because the spread is charged once per trade however wide the barriers are, so a narrow barrier spends much of its own range on costs. DeriveBarrierGeometry's own comment already said the ratio buys nothing; the objective just never followed from it. Blocker this had to solve first: m_excUpCache/m_excDownCache hold only MAXIMUM travel each way, and a maximum cannot say which side was reached FIRST - so any geometry other than the walked one was undecidable on precisely the bars where both barriers were touched, ~28% of the sample. - BARRIER_LADDER: per bar, the first-touch AGE for 8 travel distances in each direction, filled during the walk the labels already run. Cursors keep it O(1) amortised per walked bar rather than 16 comparisons. Levels are travel FROM ENTRY, not barrier prices, so one ladder serves both directions and the spread is applied analytically when a level converts back to an SL/TP multiple - storing prices would need four ladders and bake today's spread into the cache. Sized, invalidated and validity-gated with the label caches. - ReportGeometryExpectancyScan: every ladder pair priced exactly off that cache - width in ATR and in SPREADS (cost efficiency, knowable without knowing d), break-even, both base rates, the share of bars resolved inside the horizon, and EV per unit of edge. Compares the widest resolvable pair against the quantile rule's pick. MEASUREMENT ONLY - the quantile rule still chooses. Nothing here can measure d, and width buys nothing if the wider target is less predictable. Base rates are printed beside each break-even because a persistent gap is DRIFT and must not be credited to the model. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 12:59:18 -04:00
//--- FIRST-PASSAGE LADDER. m_excUpCache/m_excDownCache hold only the MAXIMUM travel each way, and a
//--- maximum cannot say which side was reached FIRST - so a candidate geometry other than the one the
//--- walk actually used is undecidable on exactly the bars where both barriers were touched, which is
//--- ~28% of the sample. Recording the first-touch AGE for a ladder of travel distances makes any pair
//--- of ladder levels evaluable exactly, with no re-walk: long wins iff its target was touched and the
//--- stop either never was or was touched later.
//--- Levels are TRAVEL FROM ENTRY (the bar's close), not barrier prices, so one ladder serves both
//--- directions and the spread is applied analytically when a level is converted back to an SL/TP
//--- multiple: a long fills at close+spread, so reaching its target needs travel = reward + spread and
//--- its stop trips at travel = risk - spread. Storing barrier prices instead would need four ladders
//--- and would bake today's spread into the cache.
#define BARRIER_LADDER_COUNT 8
const double BARRIER_LADDER[BARRIER_LADDER_COUNT] = {0.50, 0.75, 1.00, 1.50, 2.00, 3.00, 4.00, 5.00};
feat: derive the ATR multiples from measured excursions - no hardcoded geometry The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in 3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of eleven guesses is not deriving anything. WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It ranks pairings by how predictable their OUTCOME is - a question about direction. The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x. Hence the scan failing its own gate on every run, and its "winner" wandering 2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is strongly measurable, so derive the geometry from that instead. stop = q25 of measured ADVERSE travel (ordinary noise does not reach it) target = q50 of measured FAVOURABLE travel (reached ~half the time, by construction, inside the horizon) Continuous, in ATR units, superseding the enum multiples. Reachability ("target on X% of bars, stop on Y%") and the implied break-even are printed so the choice is auditable rather than trusted. FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the horizon with the target (first-passage time grows with the band) and the excursions are measured OVER the horizon, so target -> horizon -> excursions -> target is a real loop - deriving once sizes the target from travel measured under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at 3 passes, and says so if it does not settle. Does NOT create expectancy, and the log says as much: chance precision equals break-even at every geometry (m/(m+k) on both sides). It buys a target the market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio forces a target the market rarely reaches, it WARNS rather than overriding - the ratio is the user's risk policy, so the honest move is to state its cost. That is the collision that once rejected 100% of setups. Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg files written earlier today still load (their length guard finds no doubles) and a model that carries them was trained on them and never re-derives. Also fixes a message from e5ceed6 that claimed "this model resumed from disk" unconditionally - it printed above a "seeding era 0" line on a brand-new model, because the branch fires whenever the cache is not built, which is equally true before a fresh model's first prebuild. A diagnostic that misreports its own trigger is worse than one that says nothing: it gets quoted back as evidence. FORCES A FULL RETRAIN (labels change). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 12:06:25 -04:00
//--- The derivation is a FIXED-POINT ITERATION, not a one-shot. ComputeBarrierHorizonBars scales the
//--- horizon with the target (first-passage time grows with the band), and the excursions are measured
//--- OVER that horizon - so target -> horizon -> excursions -> target is a loop. Deriving once would set
//--- the target from travel measured under the OLD horizon and quietly mis-state it. Re-measure until the
//--- multiples stop moving, capped so a pathological oscillation cannot spin forever.
fix: normalise the asymmetry target - the raw one is confounded by volatility Three symbols ran the excursion test. RANGE/UP/DOWN cleared on all three; raw ASYMMETRY cleared on EURUSD and USDCAD at p=0.0050 and not on SP500 (p=0.1045). That looked like the first directional signal this project has found. It probably is not, and the test as built could not tell. (up-dn) IS NOT SCALE-FREE. If sigma is predictable - and RANGE clears at ~4x its null on every instrument - and the directional part is symmetric noise eps, then up-dn ~ sigma*eps, so a large sigma pushes the value into BOTH outer terciles. A pure volatility predictor scores positive MI against a 3-bin (up-dn) while carrying no directional information at all. Crucially that confound REPLICATES, so reproducing on two instruments is not evidence against it - and the effect sizes fit it: asymmetry runs 1.3-1.6x its null where RANGE runs ~4x, and carries ~0.1% of the target's entropy against RANGE's ~0.9%. That is the shape of a leaked fraction of the volatility signal, not an independent one. So add (up-dn)/(up+dn): bounded in [-1,+1], volatility divided out, and the only target a directional claim may rest on. The verdict now separates the cases and NAMES the confound when raw clears while normalised does not, instead of reporting the raw line as a finding. Two bugs of mine in the same block, both caught by output rather than review: - The derived-geometry line had a MISORDERED argument list: it printed "stop 25.00*ATR (q3 of adverse travel)" - the quantile percentage as the multiple and the multiple as the quantile. Real values were 2.61 stop / 8.03 target. A 25*ATR stop is absurd on its face, which is why it was seen. - THE STOP QUANTILE WAS BACKWARDS, and this one changes labels. It was 0.25 "so ordinary noise does not reach it", but q25 means 75% of bars EXCEED the stop - hit three times in four. The printed reachability said exactly that ("stop on 75.0% of bars"). Now 0.75. A quantile is a threshold, not a rate. This is the entire reason reachability is measured and printed rather than assumed. Also raises BARRIER_DERIVE_MAX_PASSES 3 -> 5: SP500 did not settle in 3 (stop still moving ~14% per pass) while EURUSD and USDCAD converged on pass 2. And bounds both quantile indices with MathMin(..., n-1) so q=1.0 cannot run off the end of the sorted array. The geometry from the previous run is NOT usable and the asymmetry result is unresolved, not established. Both are decided by the next run. FORCES A FULL RETRAIN (the stop quantile changes every label). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 13:04:13 -04:00
#define BARRIER_DERIVE_MAX_PASSES 5
feat: derive the ATR multiples from measured excursions - no hardcoded geometry The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in 3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of eleven guesses is not deriving anything. WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It ranks pairings by how predictable their OUTCOME is - a question about direction. The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x. Hence the scan failing its own gate on every run, and its "winner" wandering 2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is strongly measurable, so derive the geometry from that instead. stop = q25 of measured ADVERSE travel (ordinary noise does not reach it) target = q50 of measured FAVOURABLE travel (reached ~half the time, by construction, inside the horizon) Continuous, in ATR units, superseding the enum multiples. Reachability ("target on X% of bars, stop on Y%") and the implied break-even are printed so the choice is auditable rather than trusted. FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the horizon with the target (first-passage time grows with the band) and the excursions are measured OVER the horizon, so target -> horizon -> excursions -> target is a real loop - deriving once sizes the target from travel measured under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at 3 passes, and says so if it does not settle. Does NOT create expectancy, and the log says as much: chance precision equals break-even at every geometry (m/(m+k) on both sides). It buys a target the market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio forces a target the market rarely reaches, it WARNS rather than overriding - the ratio is the user's risk policy, so the honest move is to state its cost. That is the collision that once rejected 100% of setups. Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg files written earlier today still load (their length guard finds no doubles) and a model that carries them was trained on them and never re-derives. Also fixes a message from e5ceed6 that claimed "this model resumed from disk" unconditionally - it printed above a "seeding era 0" line on a brand-new model, because the branch fires whenever the cache is not built, which is equally true before a fresh model's first prebuild. A diagnostic that misreports its own trigger is worse than one that says nothing: it gets quoted back as evidence. FORCES A FULL RETRAIN (labels change). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 12:06:25 -04:00
#define BARRIER_DERIVE_TOLERANCE 0.05
//--- Warn when the reward:risk floor forces a target this market rarely reaches. Not a hard error: the
//--- ratio is the user's risk policy, and the honest response is to say what it costs, not to override it.
#define BARRIER_MIN_TP_REACH_PCT 20.0
diag: is "optimal SL/TP" learnable? Score the features against excursions Proposed direction: train the net to predict entry/SL/TP that maximise return and minimise drawdown, rather than to classify direction. Before rebuilding a head, measure whether the target is learnable at all. That question splits into two that behave nothing alike: HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility clustering is about the most robust regularity in markets. WHICH WAY it goes first (the asymmetry) - direction, which is what every noise-floor verdict in this project has been about. Expectancy comes ONLY from the second. The first buys position sizing and drawdown control - worth having under prop-firm limits, but not an edge: exit management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with expectancy FLAT. Crucially this is NOT already answered. Every MI figure here scored the triple-barrier label, i.e. one specific question at one fixed geometry. A noise-floor result there says nothing about whether excursion MAGNITUDE is learnable - different target, different answer. Four targets, and the verdict is the CONTRAST, printed explicitly because the dangerous misreading of "UP clears" is "we can predict profitable trades": RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that SHOULD clear. Every prior verdict here lacked a control expected to pass; a range target at the floor indicts the measurement, not the market. UP / DOWN - MFE / MAE. ASYMMETRY - up-dn, the only one that can pay. Collected inside the walk the label already does (one max, one min per bar). The early-out when both barriers resolved is GONE: it would have truncated the excursions at whichever bar tripped the last barrier, making the measurement a function of the CURRENT SL/TP - the circularity this is trying to escape. The loop was already bounded by the horizon, so only the average cost moves. Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is fat-tailed and fixed-width bins would put nearly every row in bin 0; it also pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and to the barrier label's ~1.02 instead of confounded by class balance. Two bugs fixed in this code before it ever ran, both of which would have produced a plausible quiet wrong answer rather than an error: - TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the accumulators were reset, so one bar's excursions would be cached under another bar's index. Cleared at the top now, ahead of every return. - An unresolvable bar is still flagged as labelled but carries excursions of exactly 0. Under equal-frequency binning a block of identical zeros drags the lowest cut onto zero and a third of the sample lands in one uninformative bin - a depressed score that reads as "not predictable", a false negative in the direction that would wrongly kill the idea. Rows where both excursions are zero are dropped; price cannot travel zero both ways over a whole horizon. Read-only diagnostic. No topology or label change: no retrain of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 10:22:41 -04:00
#define MI_TARGET_BARRIER 0 // shipped 3-class triple-barrier label
#define MI_TARGET_EXC_UP 1 // (maxHigh - entry)/ATR over the horizon, 3 equal-frequency bins
#define MI_TARGET_EXC_DOWN 2 // (entry - minLow)/ATR
#define MI_TARGET_EXC_RANGE 3 // up + down: pure realised volatility, the control that SHOULD clear
fix: normalise the asymmetry target - the raw one is confounded by volatility Three symbols ran the excursion test. RANGE/UP/DOWN cleared on all three; raw ASYMMETRY cleared on EURUSD and USDCAD at p=0.0050 and not on SP500 (p=0.1045). That looked like the first directional signal this project has found. It probably is not, and the test as built could not tell. (up-dn) IS NOT SCALE-FREE. If sigma is predictable - and RANGE clears at ~4x its null on every instrument - and the directional part is symmetric noise eps, then up-dn ~ sigma*eps, so a large sigma pushes the value into BOTH outer terciles. A pure volatility predictor scores positive MI against a 3-bin (up-dn) while carrying no directional information at all. Crucially that confound REPLICATES, so reproducing on two instruments is not evidence against it - and the effect sizes fit it: asymmetry runs 1.3-1.6x its null where RANGE runs ~4x, and carries ~0.1% of the target's entropy against RANGE's ~0.9%. That is the shape of a leaked fraction of the volatility signal, not an independent one. So add (up-dn)/(up+dn): bounded in [-1,+1], volatility divided out, and the only target a directional claim may rest on. The verdict now separates the cases and NAMES the confound when raw clears while normalised does not, instead of reporting the raw line as a finding. Two bugs of mine in the same block, both caught by output rather than review: - The derived-geometry line had a MISORDERED argument list: it printed "stop 25.00*ATR (q3 of adverse travel)" - the quantile percentage as the multiple and the multiple as the quantile. Real values were 2.61 stop / 8.03 target. A 25*ATR stop is absurd on its face, which is why it was seen. - THE STOP QUANTILE WAS BACKWARDS, and this one changes labels. It was 0.25 "so ordinary noise does not reach it", but q25 means 75% of bars EXCEED the stop - hit three times in four. The printed reachability said exactly that ("stop on 75.0% of bars"). Now 0.75. A quantile is a threshold, not a rate. This is the entire reason reachability is measured and printed rather than assumed. Also raises BARRIER_DERIVE_MAX_PASSES 3 -> 5: SP500 did not settle in 3 (stop still moving ~14% per pass) while EURUSD and USDCAD converged on pass 2. And bounds both quantile indices with MathMin(..., n-1) so q=1.0 cannot run off the end of the sorted array. The geometry from the previous run is NOT usable and the asymmetry result is unresolved, not established. Both are decided by the next run. FORCES A FULL RETRAIN (the stop quantile changes every label). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 13:04:13 -04:00
#define MI_TARGET_EXC_ASYM 4 // up - down: RAW asymmetry - CONFOUNDED BY VOLATILITY, see below
//--- SCALE-FREE asymmetry, and the only one of the two that can support a directional claim.
//--- (up-dn) is NOT scale-free: if sigma is predictable - and RANGE clears at ~4x its null on every
//--- instrument tested - and the directional part is symmetric noise eps, then up-dn ~ sigma*eps, so a
//--- large sigma pushes the value into BOTH outer terciles. A pure volatility predictor therefore scores
//--- positive MI against a 3-bin (up-dn) while carrying no directional information whatsoever, and it
//--- does so consistently across instruments - so replication does not rule it out. Measured 2026-08-07:
//--- raw ASYM cleared on EURUSD and USDCAD at p=0.0050 exactly where RANGE was strongest.
//--- Dividing by (up+dn) removes the scale factor and leaves the question actually being asked: given
//--- that price moved, WHICH WAY did it move further. Bounded in [-1,+1] by construction.
#define MI_TARGET_EXC_ASYM_NORM 5
fix: make the indicator tuner actually measure, and gate what it installs ROOT CAUSE of the zero spread measured on SP500 H1 2026-08-07 (all 17 candidates returned exactly 0.00359 nats): the tune loop re-inits the indicators and then scores, with no RefreshData() between. ReInitADIndicators() does its part - Create() builds a NEW handle carrying the new parameters, and the feature cache is flagged stale so features really are recomputed. But BufferTempDataCompute() reads the CIndicatorBuffer objects, and only Refresh() copies data out of a handle into those. So every candidate was scored on values still held from the PREVIOUS handle. My earlier guess in the diagnostic ("suspect the feature cache") was wrong: the cache invalidation works. Two things land together, because neither is safe alone: 1. RefreshData() after the re-init, so a candidate is scored on its own features. 2. A SELECTION GATE on the install. bestScore is a MAXIMUM over candidates, and the maximum of N draws from a null beats its incumbent almost every time - so "it beat the incumbent" installs noise. This selector is the highest-stakes of the three found in this audit because it ACTS: it overwrites the user's configured indicator settings and forces BuildFreshTopology(), so the network then trains on whatever the noise picked. Fixing (1) without (2) would have made a dormant bug actively harmful. The gate draws the winner's own permutation null once, then corrects the p-value for having chosen it out of N with Sidak: p_family = 1 - (1-p)^N. Sidak rather than the max-of-N resample used by the geometry scan because each candidate here has a DIFFERENT feature set, so their draws cannot be pooled; Sidak needs only the one null. Exact under independence, mildly anti-conservative under positive dependence - stated in the comment rather than hidden. A rejected winner restores the configured settings, which best[] cannot do since the descent mutates it. Also reports the least-ready tunable handle's BarsCalculated(). IndicatorCreate() calculates asynchronously, so if the spread is STILL zero the handles simply are not done and the tuner needs to yield between candidates rather than score them back to back - a state machine like the label prebuild. That distinction is now readable from the log instead of requiring another guess. No input, topology or label change: no retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 09:31:06 -04:00
//--- Significance the indicator tuner's winner must reach, AFTER correcting for having been chosen out of
//--- N candidates. Same 0.05 as elsewhere; what matters is that a gate exists at all, since this selector
//--- overwrites the user's configured indicator settings and forces a fresh topology.
#define MI_TUNE_ALPHA 0.05
fix: correct the lag profile across lags too - it contradicted itself 3271f1e tested each of ~21 lags against its OWN null at alpha 0.05 and starred whatever cleared. That is about one false positive per run before any signal exists, and because neighbouring lags share nearly their entire feature window the false positives arrive in CLUSTERS that read like a hump. It did exactly that on SP500 H1, twice in one afternoon on identical data: 13:55 nothing clears at any lag headline MI p=0.4478 16:22 k6/k10/k12/k16 starred, headline MI p=0.8756, observed "information survives to lag 16" BELOW its own null mean Same 31 features, same 2009 samples, same 287 blocks, cross-asset absent in both - so this was not two different measurements. Non-replication on identical data is the signature of an uncorrected multiple comparison, and acting on the second run would have pinned the lookback to 17 off noise. Galling detail: 04ee2e1 had just added exactly this correction to the barrier-geometry scan one function below. The rigorous bar went on the report with 6 candidates and the naive one stayed on the report with 21. So the lag profile now uses the same construction as the geometry winner test: one draw from every lag, keep the largest, repeat; a lag clears only by beating that distribution. Draws centred leave-one-out to match how the observed excess is centred. Independence across lags overstates the spread of the maximum (neighbours share their window), so it errs toward rejecting. Also: the positive branch now says to re-run before acting, because one run of this report has demonstrably not been a result; and MI_LAG_MAX_PROFILE caps the retained-draw matrix rather than trusting a derived m_historyBars. Read-only diagnostic. No input, topology or label change: no retrain, and a training run already in flight stays valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 16:28:57 -04:00
//--- Ceiling on profiled lags, sizing the retained-draw matrix. m_historyBars is derived and could in
//--- principle exceed this; the profile then covers the first MI_LAG_MAX_PROFILE-1 lags and says so via
//--- the lag count it prints, rather than overrunning the buffer.
#define MI_LAG_MAX_PROFILE 32
fix: gate the barrier-geometry winner on a family-wise null, not its own The scan ends by printing "set SL_Mode/TP_Mode to <winner> and retrain". That advisory fired on `bestExcess > cfgExcess * 1.5` - a ratio between two numbers, with no test that either is distinguishable from zero. bestExcess is a MAXIMUM over the eligible candidates. The maximum of several draws from a null sits well above any single draw from it, so a max-shaped statistic tested against a single-candidate null crowns a winner on noise almost every time. On SP500 H1 the winner is 2:8 at +0.00081 nats - and the lag profile committed in 3271f1e measures the pure-noise swing on this exact data at +/-0.0004, peaking at +0.00042 with nothing clearing its own null at any lag. The advisory was one ratio away from talking us into relabelling and retraining all four topologies to chase that. So build the null OF THE MAXIMUM: retain every candidate's permutation draws, take one draw from each candidate, keep the largest, repeat. The winner must beat that distribution. - draws centred LEAVE-ONE-OUT, so a draw is centred by a mean excluding it - exactly how the observed score is centred. Centring a draw by a mean that contains it shrinks it toward zero and would deflate the null. - only ELIGIBLE candidates enrol: the family the max was taken over is the family to correct for, and a clamped or sub-minRR pairing can never win. rrOK hoisted above the draws for this. - draws per candidate 20 -> MI_GEOMETRY_PERMUTATIONS (40): they now have to resolve an upper tail, which is where 20 draws are thinnest. - MI_GEOMETRY_ALPHA 0.05, stricter than the lag profile's: a wrong lookback costs input width, a wrong geometry costs a full retrain from era 0. Independence across candidates overstates the spread of the max (the real candidates share features and overlapping label windows), so the gate errs toward rejecting - the safe direction when passing costs a retrain. Read-only diagnostic. No input, topology or label change: no retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 14:04:16 -04:00
//--- Draws per candidate in the barrier-geometry scan. Raised from the ranking-only 20 because these draws
//--- now do a second job: they build the FAMILY-WISE null that decides whether the winner is real (see
//--- ReportBarrierGeometryScan). A max-statistic lives in the upper tail of the null, and a tail is exactly
//--- where 20 draws are thinnest. Cost is draws x candidates on an already-extracted sample - the whole
//--- scan ran in 3.2s at 20 draws, so this is single-digit seconds, once, before era 0.
#define MI_GEOMETRY_PERMUTATIONS 40
//--- Family-wise significance for the geometry winner. Stricter than MI_LAG_ALPHA because the two decisions
//--- are not comparable: a lag profile that guesses wrong costs some input width, whereas acting on this
//--- one means RELABELLING and retraining every topology from era 0. The gate must be hard to pass.
#define MI_GEOMETRY_ALPHA 0.05
//--- Ceiling on scanned candidates, sized to the shipped grid (2 stop multiples x 6 target multiples). Only
//--- used to size the fixed draw matrix below; the loop still skips ineligible pairings.
#define MI_GEOMETRY_MAX_CANDIDATES 12
fix(autotune): MI scorer read an array nobody filled; add the permutation floor THE TUNER WAS A SILENT NO-OP. Every chart logged auto-tune complete - 17 candidate settings scored in ~139s, feature/label mutual information 0.0000 -> 0.0000 nats (no improvement) 0.0000 is not a weak result, it is a broken measurement: finite-sample MI is biased UPWARD, so even pure noise scores above zero. Cause: ScoreCurrentParamsByMI called BufferTempDataCompute(), which APPENDS the bar's features to TempData and never touches m_featureCache - only the caching wrapper BufferTempData() writes that array. It then read m_featureCache, which ReInitADIndicators had just invalidated. Every column came back constant, FeatureColumnMI returned 0 for all of them, and all 17 candidates tied at exactly zero. 139 s per chart to return the settings it started with. Now reads the values back out of TempData, where they actually land. And an exactly-zero best score is called out as a fault rather than reported as "no improvement", because that is what it is. ADDED: a PERMUTATION BASELINE, which is the diagnostic this project has been missing. MI's finite-sample bias is ~(bins-1)(classes-1)/(2n) nats - at these sample sizes the same order as any real edge in this domain - so a raw MI figure is uninterpretable on its own. Shuffling the labels destroys every genuine association while leaving sample size, binning and class proportions intact, so the score it produces IS this dataset's noise floor, measured rather than approximated. The log now reads feature/label information - X nats against a shuffled-label floor of Y and says outright whether the features carry usable information about the target. It needs no training, no topology and no convergence, so unlike every accuracy number in this codebase it cannot be confounded by an optimizer or an objective. If the score sits on the floor, no change of architecture can help - which is the question the last three days of zero-edge results have been circling. DEPLOY FLOOR: `dirPrecPct > chancePrecPct` passed anything above chance by any amount. At ~11,000 directional calls the standard error of the precision estimate is ~0.4pp, so that gate was accepting sub-one-sigma noise - the perceptron deployed at edge +0pp on 2026-08-01. Now requires EDGE_MIN_SIGMAS (2.0) standard errors above chance, computed from the actual call count, so the bar scales with the evidence instead of needing a hand-picked constant. Recorded with it, because it is why chance is the right reference at all: under a driftless random walk P(touch +k*ATR before -m*ATR) = m/(m+k), and the break-even win rate for a k:m reward:risk trade is ALSO m/(m+k). The label's own base rate IS the break-even rate, at every SL/TP setting. So "beats chance" and "is profitable" are the same test, and no choice of SL/TP can manufacture an edge - only prediction can. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:05:50 -04:00
//--- How many standard errors a checkpoint's directional precision must clear chance by before it is
//--- considered deployable - see the edgeFloorPct block in Train(). Two sigma is the conventional
//--- "not a fluke" bar and lands near 1pp at the ~11,000 directional calls a full OOS era produces.
//--- CRITICAL CONTEXT, and the reason chance is the right reference at all: under a driftless random
//--- walk the probability of touching +k*ATR before -m*ATR is m/(m+k), and the break-even win rate for a
//--- k:m reward:risk trade is ALSO m/(m+k). The label's own base rate IS the break-even rate, at every
//--- SL/TP setting. So "beats chance" and "is profitable" are the same test, no SL/TP choice can
//--- manufacture an edge, and this margin is measuring expectancy directly.
#define EDGE_MIN_SIGMAS 2.0
fix: the Adam second moment was never Adam - all four tiers Root cause of the B=32 regression, and it predates F4 entirely. Every Adam kernel stored v already square-rooted and then fed that stored value back in as if it were the variance: v_new = sqrt(b2 * v_old + (1 - b2) * g^2) That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below unit scale, so the denominator stops tracking the gradient and Adam degrades into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll (batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where a scale-invariant optimizer gives the same distance for both. After the fix all six magnitudes read 1.199 and v tracks |g| exactly. It hit conv/LSTM specifically because they sit behind a batch-norm with running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep in the degraded regime - while the dense stack near the loss stayed in the working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/ 0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already squared v back for gamma/beta and its comment named the kernels as wrong, which is exactly why gamma/beta kept training while the stages behind froze. Persisted .nnw needs no migration - v keeps its std-dev meaning. Also, the two ways F4 exposed it, both mine: - No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods (Krizhevsky 2014; Granziol et al. 2022), applied once in InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD. - Plateau patience denominated in eras, so raising B made the ladder 32x more impatient in its only unit. PAI converged at era 41 on ~49k updates where the same config had been finding new bests at era 1028. TrainPlateauPatienceEras() stretches it by the same sqrt(B). TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23 eras per stage, not 8 -> 45). Both helpers are identities at B=1. Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into the ranking key rather than checked at deploy time so a one-sided era cannot become best-so-far in the first place. Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded CAppDialog teardown that sat ahead of it - the same ordering inversion the rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit (vs ~1.1 s for the three that finished) having reached none of its cleanup, so its arrows stayed on the chart. Steps are now timed in the log. PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now walks every object type and reports the object counts when both are zero. Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt. FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
//--- BOTH-DIRECTIONS FLOOR. A model that has stopped calling one side entirely is degenerate, and the
//--- coverage+precision test above cannot see it: coverage counts directional calls without caring
//--- that they are all the same direction, and one-sided precision can sit above chance while the
//--- model is simply riding the sample's drift. Observed 2026-08-09: the perceptron reported
//--- "Sell:0%" in every one of its 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance,
//--- deployed, and sprayed buy arrows across the chart.
//---
//--- Deliberately far below m_minDirectionalRecallPct (the 40%-per-class diagnostic): that floor is
//--- effectively unreachable on this data and was demoted for exactly that reason, so reusing it here
//--- would block every deployment rather than the pathological ones. 10% only catches a side that has
//--- genuinely gone silent, and it applies to Buy and Sell only - Neutral is the majority class and a
//--- low Neutral recall is a model taking positions, not a broken one.
#define DEPLOY_MIN_SIDE_RECALL_PCT 10.0
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
//--- FAMILY-WISE DEPLOYMENT GATE. EDGE_MIN_SIGMAS above is a PER-ERA test, and the deployed model is the
//--- MAXIMUM over every era a run produced - which is the one construction this project has repeatedly
//--- proven crowns noise. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so
//--- over N eras the chance that at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70,
//--- 93% by era 112. The gate was therefore near-certain to open on a long run no matter what the data
//--- contained, and it did - HYBRID deployed on 2026-08-08 at dir-precision 35.5% vs 34% chance, +1.5pp,
//--- selected as the best of 112 eras whose per-era values wandered between 30% and 35.5%.
//--- Every OTHER best-of-N decision in this codebase already carries this correction and every one of
//--- them REJECTS on this data: the barrier-geometry winner ("tested against the null of the maximum over
//--- 6", p=0.3902), the indicator tuner (Sidak "p_family = 1 - (1-p)^N", p=1.0000), the MI lag profile
//--- ("against the null of the MAXIMUM over 21 lags"). The one decision that actually ships a model to a
//--- live account had none. See BestCheckpointSurvivesSelection().
//--- Sidak on the era count, matching the tuner's idiom. NOTE it is CONSERVATIVE here: consecutive eras
//--- share the same OOS bars and differ by one gradient step, so they are nowhere near N independent
//--- draws and the true family-wise error is below this bound. Erring strict is deliberate - this gate
//--- decides what trades real money, and the codebase's whole posture is reject-unless-demonstrated.
#define DEPLOY_FAMILY_WISE_ALPHA 0.05
refactor(ai): nine class-imbalance inputs down to two The imbalance section offered nine controls for one job. Audited against the code, five of them did not do what their names said at the shipped defaults: AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns whenever the adjusted loss is on, which is default. OversampleParity DEAD in training - Training.mqh gated the replay loop on !useLogitAdjustedLoss (correctly, citing Buda et al. 2018). Live only in the online-learning path. EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma damper - "replay minority bars through pass-2 oversampling" was a focal-loss switch. ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25. UseStaticPrior An exact duplicate of FreezePriorCalibration - the two were OR'd together in the single place either is read. So they were not five mechanisms fighting; they were one mechanism plus eight knobs that mostly described machinery that no longer ran. That is worse than a real conflict, because the log agreed with the names: the label-cache line printed "reps up to 28x (90% parity) (seeding era 0's class-balance oversampling)" on every run, describing an oversampling pass that had been switched off. It is fixed here too - it cost this session a wrong diagnosis. The one genuine redundancy was focal loss, running at gamma*0.125 alongside the adjusted loss: two corrections on the same axis, the exact stacking failure this file already cited Buda et al. for in two other places, damped by a replay flag whose replay path was itself dead. Removed rather than re-tuned. The plateau ladder is unaffected - its escape is the learning-rate warm restart; the gamma anneal beside it only ever stepped toward zero. WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze: LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted- Loss boolean, since a strength dial where 0 already means off does not need an on/off switch beside it. FreezePriorCalibration unchanged. It is the only one of the six corrections with a consistency guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection already ranks on - so the loss and the deploy decision optimize one thing. The online continual-learning path keeps its own alpha-balanced focal weight, now as constants pinned to the removed inputs' shipped defaults, so its behaviour is unchanged. It legitimately needs its own correction: ApplyLogitAdjustment() only runs inside a training run, so a deployed model that was reloaded carries no logit offsets and would otherwise stream 31:1 data into itself uncorrected. The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a double fed to a %d conversion and had always emitted a literal 0; the |MR: segment is written as the constant its shipped defaults produced. Dropping either would have re-keyed every model and forced a from-scratch retrain of the one topology currently converged and trading. Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable "neutralized by prior correction" diagnostic. Both builds compile 0 errors, 0 warnings. No retrain forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- OVERSAMPLING CONSTANTS REMOVED 2026-07-31 (MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION).
//--- Data-level class-balance oversampling is gone; every bar is queued exactly once and the imbalance
//--- is corrected analytically in the gradient by the logit-adjusted loss. See the queueing block in
//--- Expert\AIBase\Training.mqh for the four successive oversampling designs that collapsed before it,
//--- and the class-imbalance audit in Variables\Inputs.mqh for the nine inputs this consolidated.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- TRIPLE-BARRIER LABELS (Lopez de Prado, "Advances in Financial Machine Learning", ch. 3).
//--- 2026-08-01: replaced exact-pivot ZigZag labels. ZigZag REMAINS the input-feature source
//--- (m_useSwingContext) and the horizon source below - only the TARGET changed.
//---
//--- Why. The old label marked the single exact bar where a ZigZag pivot confirmed: ~1,164 Buy, ~1,164
//--- Sell, ~35,841 Neutral - a 31:1 imbalance, and every correction mechanism in this file (logit
//--- adjustment, prior EMA, bias seed, recall floor, alternation gate, NMS, four oversampling designs)
//--- was downstream of that one choice. The imbalance was self-inflicted by the target, not a property
//--- of the market: the reference book (references\neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag
//--- but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50, nothing to correct.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//---
//--- What replaces it. For each bar, place the EA's OWN stop and target around a hypothetical entry at
//--- that bar's close and ask which barrier a real trade would touch first, within a horizon:
//--- long reaches TP before SL, short does not -> Buy
//--- short reaches TP before SL, long does not -> Sell
//--- neither resolves in the model's favour -> Neutral
//--- The barriers come from SL_Mode/TP_Mode (m_sl_mode/m_tp_mode, inherited from CExpertSignalCustom),
//--- so the label and the trade CANNOT drift apart, and `dir-precision` in the era line stops being a
//--- proxy and literally becomes the win rate of the strategy under its own exit rules. That is the
//--- number this project never had. No new user input: the two that shape the label already exist.
//---
//--- Expected balance. At the shipped SL_ATR_x1 / TP_ATR_x3 the gambler's-ruin probability of touching
//--- +3 ATR before -1 ATR is 1/(1+3) = 25% per side, so roughly 25/25/50 - about 2:1 rather than 31:1.
//--- Measured for real at the end of the prebuild; do not assume it.
//---
//--- INTRABAR AMBIGUITY IS RESOLVED PESSIMISTICALLY. When one bar's range spans both barriers, OHLC
//--- cannot say which came first, so the label counts it as the STOP. A win rate built on the
//--- optimistic reading is exactly the kind of number that evaporates live.
#define BARRIER_TIE_GOES_TO_STOP 1
//--- Vertical (time) barrier, in bars. DERIVED, never configured: the median distance between confirmed
//--- ZigZag pivots over the training window - i.e. this symbol/timeframe's own natural swing horizon,
//--- measured from the same indicator the features already read. Snapped to the ladder below so the
//--- estimate has to move ~30% to change the answer; without that quantization a horizon that drifted as
//--- history downloaded would silently relabel a partially-trained model's targets mid-run. Same
//--- measure-once-then-quantize contract as ComputeFirstLayerWidth().
//--- It is deliberately NOT in the weights-filename fingerprint - a filename keyed on a measured
//--- quantity orphans a trained model the moment the measurement moves. See BuildConfigFingerprint.
fix(labels): the 128-bar horizon ceiling was truncating the shipped label The corrected geometry scan exposed something bigger than the geometry question it was asked. Every pairing from 2:6 upward came back CLAMPED - including 2:6, the SHIPPED configuration. First-passage time for a driftless walk leaving [-m,+k] goes as m*k, and the measured swing median here is ~12 bars at m*k=1, so 2:6 wants ~144 bars and 3:10 wants ~360. The ladder stopped at 128. A clamped label stops meaning "does the target come before the stop" and quietly becomes "...within 128 bars", while the deployed EA holds until SL or TP with no bar limit. So the target the models have been trained on all along was not the strategy the EA executes, and the trades it silently reclassified as Neutral were the SLOW WINNERS - precisely the ones a 1:3 barrier exists to capture. Timeout share stayed ~0% throughout, which is why this never showed up: the truncation lands in Neutral, not in the timeout counter that was watching for it. Ladder extended to 384 (12..128, 192, 256, 384) so every selectable geometry gets an honest horizon. Cost is one embargo of at most 384 bars out of ~38k. Second fix, same class of error as the H(Y) one: the scan's "best eligible" was 2:2, a 1:1 barrier, against a shipped Min_Risk_Reward_Ratio of 1:2. Training four topologies on that target would have produced a model whose every setup is rejected at the door - the exact failure behind four consecutive Market rejections for "no trading operations". Sub-minRR geometries are now ineligible and marked [<minRR], printed rather than hidden. Also drops the dense-depth tag from the display name ("Perceptron 3L" -> "Perceptron"). Depth is derived, so it names nothing a user chose; the config tag [PAI-0be2] already disambiguates concurrent charts and does it for every input rather than one. Full topology still logged by "config -". Compiles 0 errors / 0 warnings, standard and Market. Build tag horizon-384-v1. Changes the LABEL for every geometry, so the next scan supersedes the previous numbers - and a retrain is required before any model trained under the truncated target means anything. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:54:45 -04:00
//--- 2026-08-01: the ladder used to stop at 128 and that ceiling, not the barrier geometry, was the
//--- binding constraint. First-passage time for a driftless walk leaving [-m, +k] is proportional to m*k,
//--- and the measured swing median here is ~12 bars at m*k=1 - so EVERYTHING from 2:6 upward wanted more
//--- than 128 bars and got truncated, including the SHIPPED 2:6 configuration. A truncated label stops
//--- meaning "does the target come before the stop" and quietly becomes "...within 128 bars", while the
//--- deployed EA holds until SL or TP with no bar limit. That is a train/deploy mismatch in the target
//--- itself, and it silently converted the slow winners - exactly the trades a 1:3 barrier exists to
//--- catch - into Neutral. Extended to cover the whole selectable grid: 3:10 needs ~360 bars.
//--- Cost is one embargo of BARRIER_HORIZON_MAX bars out of ~38k, i.e. nothing.
#define BARRIER_HORIZON_LADDER_COUNT 11
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
#define BARRIER_HORIZON_MIN 12
fix(labels): the 128-bar horizon ceiling was truncating the shipped label The corrected geometry scan exposed something bigger than the geometry question it was asked. Every pairing from 2:6 upward came back CLAMPED - including 2:6, the SHIPPED configuration. First-passage time for a driftless walk leaving [-m,+k] goes as m*k, and the measured swing median here is ~12 bars at m*k=1, so 2:6 wants ~144 bars and 3:10 wants ~360. The ladder stopped at 128. A clamped label stops meaning "does the target come before the stop" and quietly becomes "...within 128 bars", while the deployed EA holds until SL or TP with no bar limit. So the target the models have been trained on all along was not the strategy the EA executes, and the trades it silently reclassified as Neutral were the SLOW WINNERS - precisely the ones a 1:3 barrier exists to capture. Timeout share stayed ~0% throughout, which is why this never showed up: the truncation lands in Neutral, not in the timeout counter that was watching for it. Ladder extended to 384 (12..128, 192, 256, 384) so every selectable geometry gets an honest horizon. Cost is one embargo of at most 384 bars out of ~38k. Second fix, same class of error as the H(Y) one: the scan's "best eligible" was 2:2, a 1:1 barrier, against a shipped Min_Risk_Reward_Ratio of 1:2. Training four topologies on that target would have produced a model whose every setup is rejected at the door - the exact failure behind four consecutive Market rejections for "no trading operations". Sub-minRR geometries are now ineligible and marked [<minRR], printed rather than hidden. Also drops the dense-depth tag from the display name ("Perceptron 3L" -> "Perceptron"). Depth is derived, so it names nothing a user chose; the config tag [PAI-0be2] already disambiguates concurrent charts and does it for every input rather than one. Full topology still logged by "config -". Compiles 0 errors / 0 warnings, standard and Market. Build tag horizon-384-v1. Changes the LABEL for every geometry, so the next scan supersedes the previous numbers - and a retrain is required before any model trained under the truncated target means anything. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:54:45 -04:00
#define BARRIER_HORIZON_MAX 384
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- Fallback when the ZigZag scan finds too few pivots to take a median from (a cold history buffer, or
//--- a symbol so quiet that Depth-12 emits almost nothing). Mid-ladder, and it logs when it fires.
#define BARRIER_HORIZON_FALLBACK 32
//--- Minimum confirmed pivots required before the median is trusted rather than the fallback.
#define BARRIER_HORIZON_MIN_SAMPLES 20
//--- Share of bars one class must hold before the era-0 output-bias cold-start seed fires - see the
//--- trigger in AdvanceLabelCachePrebuild(). A +-3.0 bias seed is a correction at a 94%-Neutral prior
//--- and a distortion at a 50% one, so the threshold marks where "dominant" actually starts.
#define COLD_START_SEED_MIN_DOMINANCE 0.70
//--- Reduce-on-regression learning-rate decay. `eta` (AI\Network.mqh) is a plain global read fresh
//--- by every weight-update call on every backend (native/OpenCL/CPU-DLL/DirectML-DLL alike), so
//--- shrinking it here takes effect on the very next backProp() everywhere at once. A fixed step
//--- size that is large enough to escape a bad random init quickly (see the sharp era 5->14 OOS
//--- accuracy climb this was tuned against) is, by the same token, large enough to overshoot once
//--- training gets close to a good solution - the era 14->16 regression right after that climb is
//--- the classic signature of that overshoot, not a structural bug. The existing best-checkpoint
refactor: compose topologies from named stages; drop dead code DRY - topology construction --------------------------- CSignalCONV and CSignalHYBRID each built the Conv+Pool front-end from scratch; CSignalLSTM and CSignalHYBRID each built the LSTM stage from scratch. The duplicates had already drifted: HYBRID guarded the LSTM step with MathMax(1, historyBars/2), CSignalLSTM divided unguarded, so a historyBars of 1 gave two different steps for what is documented as the same layer. Extracted AddConvPoolStage() and AddLstmStage() onto CExpertSignalAIBase. The three overrides are now compositions: CONV = AddConvPoolStage LSTM = AddLstmStage HYBRID = AddConvPoolStage && AddLstmStage HYBRID's "matches the standalone CONV front-end exactly, then adds LSTM" is enforced by construction instead of by comment. Took the guarded step for both. Also fixed a descriptor leak the duplicates shared: on a failed topology.Add() the CLayerDescription was neither owned by the array nor deleted. Dead code --------- - CNet::SaveCheckpoint / CNet::LoadCheckpoint (123 lines). Superseded by the in-memory CaptureWeights/RestoreWeights pair; Network.mqh:1312 already said so ("This replaces the file-based SaveCheckpoint/LoadCheckpoint"). Zero call sites - every remaining mention was a comment. The five comments that referenced them have been reworded rather than left dangling. - CExpertSignalCustom::CheckForDuplicateTrade / FindLastTradeIndex / UpdateTradeStatusAndExit: declared, never defined anywhere, never called. They only made it look as though duplicate-trade detection existed. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:38:05 -04:00
//--- mechanism just below (m_bestOosForecast + CNet::CaptureWeights) already guarantees the FINAL deployed
//--- model can't regress; this constant lets the live training process itself settle down instead
//--- of continuing to oscillate around a good solution once it finds one.
#define ETA_DECAY_REGRESSION_PCT 5.0 // only decay after a real regression, not per-era noise
#define ETA_DECAY_FACTOR 0.7
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//--- 1e-5, not the old 1e-4: with the ceiling at 3e-4 the old floor gave the whole decay schedule a
//--- 3x dynamic range - three 0.7x decays and it was pinned, so "reduce LR on regression" could never
//--- actually settle a run that kept oscillating (2026-08-09 audit, F2). 30x leaves the schedule room
//--- to genuinely calm down; the recovery bump still climbs back at 1/0.7 per new best, so a run that
//--- resumes improving is not stuck crawling.
#define ETA_MIN 0.00001
fix(training): escape the recall-gate catch-22 that let runs decay unchecked Evidence (MQL5\Logs, SP500 H1, 2026-07-29): Perceptron era 61 Buy 32% Sell 27% Neut 94% bal 51% LSTM era 160 Buy 16% Sell 11% Neut 98% bal 42% (peaked 49% @ era 44) Hybrid era 179 Buy 5% Sell 2% Neut 99% bal 35% (peaked 41%) CONV era 228 Buy 2% Sell 4% Neut 99% bal 35% (peaked 40% @ era 122) Every model peaks early then decays monotonically toward Neutral, and nothing stops it: the restore-best-weights + decay-eta handler is gated on m_bestPassedRecall, which stays false forever when no checkpoint ever clears the per-class floor. CONV ran 228 eras with eta pinned at its 0.000300 start. The plateau ladder cannot end such a run either (stage 3 refuses to deploy without a recall pass, so it resets ~27 times), making it a 1000-era one-way trip. The gate's own justification had expired. It was written when the pre-pass tiebreak was blended-accuracy-only, where "best" really did mean "called Neutral most confidently". The balanced-selection change replaced that with `balancedOosEra > m_bestBalancedOos` plus an isFullyCollapsedEra exclusion, so a Neutral-only era now scores ~33% - the FLOOR of the balanced metric - and cannot anchor the checkpoint at all. Pre-pass "best" now means "most class-balanced so far", which is worth defending; and isWorseEra is itself a balanced-accuracy regression, so it cannot fire merely for trading Neutral calls for Buy/Sell. The original concern still holds while the best-so-far IS near-collapse, so the escape is margin-guarded: defend the checkpoint only once balanced accuracy sits more than BALANCED_WORTH_DEFENDING_MARGIN_PCT (5pp) above the one-class floor of 100/3. Against the run above that engages for all three stuck topologies (42.3/41.3/50.0 vs a 38.3 threshold) while a genuinely collapsed run still explores freely. Two inputs restored to the regime that actually produced a deploy: - MinRecall 60 -> 40. The one successful auto-deploy in the logs (Hybrid, 28th 00:50, best balanced 66.0%) ran against a 40% floor. 60 has never been shown reachable here - a floor above what the config can reach is the same "target set too high" failure the surrounding comment already warns about. - OversampleParity 60 -> 90. 60 overcorrected. Runs now START Neutral-dominant (Buy 0-11% recall at era 1) and call Buy/Sell on 0-4% of bars against a ~6% true base rate - under-calling, with no headroom to converge down from. The deploying run began at Buy 90% / Sell 36%, 24% of bars called, and settled into the floor from above. Raw over-calling is the intended starting condition; live calls are base-rate-calibrated by AILogitPriorStrength, which is why the input's own note says to judge over-calling by live-fired precision, not raw counts. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 07:51:08 -04:00
//--- Balanced accuracy (macro-recall) of a model that puts every bar in ONE class: (0+0+100)/3. It is
//--- the FLOOR of the balanced metric, not a midpoint - any genuinely multi-class model scores above it.
refactor(ai): derive the first dense layer's width instead of asking for it InitialNeurons was an input whose only defensible value depends on two things the user cannot see when picking from a dropdown: how wide the input vector ended up after feature selection, and how much in-sample data the study period actually yields. Left to a hand-picked constant it was badly wrong - 500 units against a 420-wide input is 210,500 weights, 72% of a 292,583-weight model, against ~36,500 training bars of which only ~2,236 are directional. That is 6.6 weights per training bar, and it EXPANDS a set of highly correlated inputs rather than compressing them. The symptom was already in the logs and had been read as a depth problem: the shallowest topology consistently beat the deepest (perceptron 52.7% balanced, hybrid 41.3%). Over-parameterization predicts that ordering just as well as covariate shift does, and only one of the two had been addressed. ComputeFirstLayerWidth() budgets roughly one first-layer weight per in-sample bar. Measured across the configurations in use: M15 10y -> 256 units, 129,071 weights, 0.73 per bar H1 10y -> 64 units, 28,727 weights, 0.65 per bar H4 10y -> 16 units, 7,559 weights, 0.68 per bar Two design points that matter: - It estimates in-sample bars from the STUDY PERIOD and timeframe, not from Bars(). What is downloaded grows over a terminal's lifetime, and a topology that widened as history filled in would re-key its own weights file and discard a trained model. - The result is snapped down to a coarse power-of-two ladder, so the estimate would have to be wrong by ~2x to change the answer. Every field it reads is already part of the weights-filename fingerprint, so the derived value needs no fingerprint entry of its own. The public setter is removed - it could only have been called after construction, and would either be ignored or silently re-key the model mid-run. Where the data cannot support even the floor (D1 over 10 years is under 2,000 bars) it now says so and names the fixes, rather than quietly training a model with more weights than examples. The DB config fingerprint drops the term too, which re-keys existing pattern databases once - correct, since a model an order of magnitude smaller should not inherit the old one's win-rate history. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 13:01:16 -04:00
//--- ComputeFirstLayerWidth() constants. SECONDS_PER_YEAR is the mean Julian year (365.25 days), the
//--- same convention MQL5's own date arithmetic uses. MARKET_OPEN_FRACTION allows for closed hours and
//--- weekends - ~0.72 is right for both a 24/5 FX week and an index with extended sessions, and the
//--- ladder in that function makes anything in the 0.6-0.85 range land on the same rung anyway.
//--- FIRST_LAYER_MIN_WIDTH is a floor for degenerate configs (a very short study period, or a symbol
//--- whose history cannot support the requested window) - below this the taper has nothing to work with.
#define SECONDS_PER_YEAR 31557600.0
#define MARKET_OPEN_FRACTION 0.72
fix(ai): cap logit-adjustment strength to the head's usable logit range tau=1.0 inverted the collapse instead of curing it. The head is SIGMOID, so each output is bounded to [0,1] and the widest logit gap the net can express between two classes is CLASS_LOGIT_SCALE * (1-0) = 6. The offsets are tau*log(prior_c), whose spread on this 30:1 imbalance is 3.42 - so tau=1.0 spent 57% of the ENTIRE expressible range on the prior correction. The network did the only thing available to it: saturate Buy/Sell outputs to 1.0 to overcome a -3.42 training handicap. The offsets are absent at inference, so that surplus made every bar directional. Measured across all five still-training charts: Neutral recall 0%, directional calls on ~100% of bars, win rate 5-7% against a ~6% base rate - no information whatsoever - while balanced accuracy read a flattering 58-64% because two of its three terms sat near 95%. OOS accuracy 6%. Menon et al. assume an unbounded logit head where a 3.42 shift is negligible against the reachable range. It is not negligible here, so the strength is now expressed RELATIVE to the range actually available: tau_eff = min(tau_cfg, LOGIT_ADJUST_MAX_RANGE_FRACTION * SCALE / spread) At 20% that gives tau 0.35 on this data. Deliberately a fraction rather than a tau ceiling: it stays correct if CLASS_LOGIT_SCALE changes, if the head becomes unbounded, or on any symbol whose imbalance differs. The input remains effective below the cap, so dialling it down needs no rebuild. Simulated at a signal strength where the task is genuinely learnable, the precision/recall frontier is monotone: tau 1.0 -> 49.6% call rate at 6.4% precision (base rate 6.1%, i.e. worthless); tau 0.35 -> 2.0% at 15.5%; tau 0.15 -> 0.2% at 33.3%. The capped value lands in the same regime the pre-logit-adjustment run occupied (1-6% of bars at 20-35% win rate). Also logs the measured priors, the spread, and whether the cap bound. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 23:20:07 -04:00
//--- Ceiling on how much of the classification head's LOGIT RANGE the logit-adjustment offsets may
//--- consume, as a fraction. The head is SIGMOID, so each output is bounded to [0,1] and the widest
//--- logit difference the net can express between two classes is CLASS_LOGIT_SCALE * (1 - 0) - six,
//--- at the shipped scale. The offsets are tau*log(prior_c), whose spread on a 30:1 imbalance is
//--- log(0.939) - log(0.031) = 3.42, so an untamed tau=1.0 spends 57% of the ENTIRE expressible range
//--- on the prior correction alone. Measured 2026-07-29: every chart did the only thing it could -
//--- saturate its Buy/Sell outputs to 1.0 to overcome a -3.42 handicap during training - and since the
//--- offsets are absent at inference, that surplus made EVERY bar directional. Neutral recall 0%, calls
//--- on ~100% of bars, win rate 5-7% against a ~6% base rate: no information at all, while balanced
//--- accuracy read a flattering 64% because two of its three terms were ~95%.
//--- Menon et al. assume an unbounded logit head, where a 3.42 shift is negligible against the range
//--- the network can reach. It is not negligible here, so the strength has to be expressed relative to
//--- the range actually available. Deliberately a FRACTION rather than a tau ceiling: it stays correct
//--- if CLASS_LOGIT_SCALE changes, if the head becomes unbounded, or on any other symbol/timeframe
//--- whose class imbalance differs - none of which a hardcoded tau would survive.
#define LOGIT_ADJUST_MAX_RANGE_FRACTION 0.20
feat(ai): rank checkpoints on directional precision, not balanced accuracy Balanced accuracy is maximized by exactly the model this system must never deploy. Measured frontier at fixed signal strength, base rate 6.1%: tau 0.00 -> calls 0.2% of bars at 27.3% precision, balanced 34.0% tau 0.35 -> calls 2.0% of bars at 15.5% precision, balanced 36.3% tau 1.00 -> calls 49.6% of bars at 6.4% precision, balanced 53.5% It rises monotonically as the model calls MORE and is right LESS, because two of its three terms are directional recalls that a call-everything model drives to ~95%, while the Neutral term it sacrifices counts for only a third. The 2026-07-29 run landed exactly there: balanced 58-64% while calling a direction on ~100% of bars at a 5-7% win rate against a ~6% base rate. Only the per-class recall floor stopped those deploying - a guard doing the job the objective should have been doing - and that same guard also rejected the genuinely useful sparse-but-precise checkpoints. Ranking is now DIRECTIONAL PRECISION: of the bars called Buy or Sell, how many were right. That is what a trading edge is. Two anti-degenerate floors bracket it, since precision alone is trivially maximized by calling almost nothing: coverage must reach a fraction of the true directional base rate (derived, not configured - it adapts to any symbol/timeframe/label rule), and precision must at least beat that base rate. Against the same frontier the deploy order inverts from tau 1.00 > 0.50 > 0.35 > 0.15 (old, worst model first) to tau 0.35 > 0.50 > 1.00 (new; 0.00/0.15 rejected on coverage) Balanced accuracy is kept in the log as a diagnostic and marked as such, so a run where the two disagree - the signature of an over-caller - is visible at a glance. MinRecall no longer decides what ships; it now only drives the diagnostic recall line and is a candidate for removal. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:13:08 -04:00
//--- Minimum directional call rate a checkpoint must reach to be considered deployable, expressed as a
//--- FRACTION OF THE TRUE DIRECTIONAL BASE RATE rather than an absolute percentage - a model that calls
//--- a direction a quarter as often as one actually occurs is sparse but usable; one that calls ten
//--- times a decade is not, however precise those ten calls were. Derived rather than configured so it
//--- adapts to any symbol/timeframe/label rule without a second input to keep in sync.
#define MIN_COVERAGE_FRACTION_OF_BASE_RATE 0.25
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//--- DIRECTIONAL CONFIDENCE THRESHOLD (2026-08-09). The training loss and the checkpoint-selection
//--- metric want different things, and until now only the second one knew it. Logit-adjusted
//--- cross-entropy pushes the head to call every class at roughly its adjusted prior - it has no term
//--- for "how often should I trade" - so the model calls a direction on 87-91% of bars. The selection
//--- metric meanwhile is precision x coverage credit, saturating at the coverage floor: above the
//--- floor, extra calls earn NOTHING and only precision counts. So selection wants few, good calls and
//--- the loss produces many mediocre ones, and all selection could do was pick the least-bad era out of
//--- whatever the loss happened to hand it. Nothing pushed the model toward selectivity.
//---
//--- This closes that gap without touching the loss (which is doing its own job correctly - it is
//--- estimating class probabilities, and a probability estimate should not be distorted to encode a
//--- trading policy). The decision RULE gets the policy instead: emit a directional call only when the
//--- softmax margin between the winning direction and its best rival clears a threshold, and pick that
//--- threshold to maximise precision subject to still clearing the same coverage floor the selection
//--- metric uses. Standard cost-sensitive-decision practice (Elkan 2001): estimate probabilities, then
//--- choose the operating point separately.
//---
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- FITTED ON A HELD-OUT CALIBRATION SLICE, APPLIED TO OOS AND LIVE. It is not enough for the fit to
//--- avoid the graded data (it always did - pass 3 comes after): the curve it reads must also be one the
//--- weights have not MEMORIZED, and harvesting it from pass 2's own backprop samples failed that second
//--- requirement badly. Measured 2026-08-10 by pairing every fit against the same era's OOS result:
//---
//--- PAI era 1 IS 25% coverage @ 66.1% win (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp
//--- PAI era 24 IS 66% coverage @ 76.4% win (+9.5pp) -> OOS 65% (-2pp) gap +11.4pp
//--- PAI era 76 IS 90% coverage @ 79.6% win (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp
//--- LSTM era 9 IS 77% coverage @ 81.6% win (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp
//---
//--- The gap grows monotonically with era while OOS stays flat, i.e. the IS curve stops describing the
//--- model's behaviour on unseen bars within a handful of eras. That is fatal HERE specifically, because
//--- FitDirConfThreshold's objective branches on the SIGN of (p - break-even): the memorized curve says
//--- +12pp at 95% coverage, so `coverage x (p - p0)` correctly maximises coverage and returns ~0.02 -
//--- fire on every bar. The `p < p0 -> get more selective` branch, which is the actual regime and the
//--- entire point of 983a6a3, could never fire because IS never showed p < p0.
//---
//--- So the slice below is carved out of the IS span, purged from backprop on BOTH sides by one label
//--- horizon, and scored after pass 2 has finished training. It costs DIR_CONF_CALIB_PCT_OF_IS of the
//--- training data. That is worth paying for a reason beyond honesty: the deploy gate needs
//--- dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero DILUTES any edge that is
//--- concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward
//--- chance by construction. A threshold that can actually be selective is the only mechanism by which a
//--- small, concentrated edge could ever clear that gate.
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//---
//--- Bin count is a resolution/noise trade-off: the margin lives in [0,1], so 50 bins put each candidate
//--- threshold 0.02 apart, fine enough to sit near the precision peak and coarse enough that each bin
//--- still holds thousands of the ~38k IS bars.
#define DIR_CONF_THRESHOLD_BINS 50
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- Below this many directional calibration calls the histogram is too sparse to choose an operating
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//--- point from, and a threshold fitted on a handful of bars is just the best-of-N error at a smaller
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- scale. The model then KEEPS THE PREVIOUS ERA'S THRESHOLD rather than falling back to 0.0: "trade
//--- every bar" is the most dangerous setting in the range, so it must never be what a failed
//--- measurement decays to. At era 0 the previous value is 0.0 anyway, so the first-era behaviour is
//--- unchanged.
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
#define DIR_CONF_MIN_FIT_CALLS 200
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- Share of the IS span held out to fit the operating point on. 15% of a ~38k-bar IS span is ~5.7k
//--- bars, ~28x DIR_CONF_MIN_FIT_CALLS even before allowing that ~99% of bars resolve directionally
//--- under the measured geometry - so the fit is never sparse, and each of the 50 bins still holds
//--- enough calls to be a rate rather than a coin flip. Larger buys precision in the fit at a direct
//--- cost in training data; smaller makes the operating point itself noisy, which is the failure this
//--- whole mechanism exists to avoid.
#define DIR_CONF_CALIB_PCT_OF_IS 15
feat: excursion-size head (Stage 1, measurement only) Direction is closed - normalised asymmetry fails on three instruments with a working positive control, and the classifier's own best-of-999 era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is a different question and RANGE clears at ~4x its null. Checked the denomination before building on that, since the source memo warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is predictable" is a claim about travel RELATIVE to current ATR, not a restatement of "ATR is autocorrelated". It is exactly the part a fixed multiple (stop 3.31*ATR, target 1.64*ATR) discards. A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches ladder rung k) upward and downward. Survival parameterisation rather than regressing the multiple, because it needs nothing new from CNet: sigmoid outputs and the per-neuron delta the `total != 3` branch already applies (a quantile head would need a linear activation and a pinball gradient in Network.mqh, Network.cl and the DirectML path, on a class four topologies share). Targets are free - m_ladderUpAt already records first-touch age per rung with 0 meaning never reached. Separate net, not extra outputs on the classifier: more outputs would change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push the count off 3 - the exact condition backProp uses to select the joint softmax gradient the 3-class head depends on. The classifier is bit-for-bit unaffected and this is removable without trace. STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the constant per-rung base rate - the baseline a fixed ATR multiple already assumes - with both predictors fitted IS and evaluated OOS, so neither gets a look at the test set. Positive skill justifies Stage 2 (drive SL/TP and sizing off ExcursionQuantile, which is defined and deliberately uncalled). Zero or negative means ATR already carries everything and Stage 2 must not be built. Trains only on primary occurrences: the replay queue oversamples for CLASS balance, and a direction-balanced sample is a biased SIZE sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
//--- EXCURSION-SIZE HEAD (see Expert\AIBase\Excursion.mqh for the whole rationale). Hidden width is
//--- deliberately small: the question it asks has a known low-dimensional answer (volatility
//--- clustering), and era time here is taken from a classifier already at ~300 s/era.
#define EXCURSION_HIDDEN_UNITS 24
//--- Below this many held-out bars the Brier skill score is noise and no verdict is printed.
#define EXCURSION_MIN_SCORED 500
//--- Skill (percent) the head must beat before Stage 2 is justified. Not zero: replacing a constant
//--- that cannot fail with a learned quantity that can needs to buy more than a rounding error, and a
//--- Brier skill under a couple of percent is inside the era-to-era wobble of the estimate itself.
#define EXCURSION_SKILL_USEFUL_PCT 2.0
fix: the excursion gate would have passed Stage 2 on an artifact I made Second-opinion review killed the +4.2% far-rung result, correctly, and the mechanism is my own bug. A head trained toward {0.05,0.9} converges to 0.05+0.85p, so its bias is 0.05-0.15p: negative where p is near 1, POSITIVE where p < 1/3, growing monotonically as the rung gets farther. Against a baseline frozen at the IS rate, an upward-biased head scores positive Brier skill whenever the OOS rate merely sits above the IS rate. Predicted signature: huge negatives near, ~zero at p=1/3, growing positives far. Observed: -82% ... -0.6% ... +1.2/+2.7/+4.2. The far rungs were not the clean end of a distorted measurement, they were the other face of the same artifact. Everything before 25aca83 is void. The gate was a bare `skill >= 2%` point estimate over 8 rungs x 4 topologies x N eras, reported per era - a best-of-~300 with no interval and no multiplicity control, which is the shape of the four traps already documented here. It now needs FOUR things at once: DECISION RUNGS only the rungs ExcursionQuantile actually reads at the live geometry (target 1.62, stop 3.31 ATR), fixed before looking. Skill at 5 ATR is skill about a distance no order is placed at - and the TARGET side currently interpolates 1.5/2.0, which measured -2.2% and -1.3%. DISJOINT SAMPLE one bar per horizon. Adjacent bars share 63 of 64 horizon bars, so ~16k scored bars is ~250 independent ones and every SE over the full set is ~8x understated. VS ORACLE the best constant achievable ON THE SCORED BLOCK, closed form from H and n (Brier = H*(1-H/n)). A head that learned only a LEVEL nearer the OOS rate than the frozen IS constant scores positive against the old baseline and <= 0 here. This is the control that separates per-bar skill from base-rate drift. MONOTONE CURVE P(reach k) must be non-increasing in k. Nothing constrained 8 independent sigmoids to obey that, and ExcursionQuantile returns the FIRST crossing - so a tangled curve is misread exactly where the head is least sure. Counted and reported, not silently used. The pass message now also states what a pass would and would not buy: expectancy is -costs at zero directional edge whatever the stop distance, and under prop DD limits LOWER variance also lowers P(reach target before limit), so "better drawdown" is a choice of failure mode, not a win. Still owed before any Stage 2: a race against a trailing-quantile incumbent and a vol-feature logistic. Beating a frozen global constant is the weakest admissible bar for replacing a global constant. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:49:32 -04:00
//--- Minimum DISJOINT (non-overlapping-horizon) observations before that tally is allowed to decide
//--- anything. ~16k scored bars over a 64-bar horizon leaves ~250 independent ones, which is the real
//--- sample size; below this the disjoint skill is noise quoted to one decimal place.
#define EXCURSION_MIN_DISJOINT 200
//--- Share of bars whose predicted survival curve is non-monotone. Above this the 8 sigmoids are not
//--- describing one distribution and ExcursionQuantile's first-crossing read is not well defined.
#define EXCURSION_MAX_MONO_VIOL_PCT 5.0
feat: race the excursion head against a trailing-quantile incumbent Beating a frozen global constant is the weakest admissible bar for replacing a global constant. The honest incumbent is a rolling rung frequency: it adapts to the volatility regime - exactly what the head claims to predict - and needs no model, no 760 inputs and no training. Implemented as a ring of per-bar outcome bitmasks (32 rungs fit one ulong), sized horizon + EXCURSION_TRAIL_WINDOW. The newest `horizon` entries are held back UNRESOLVED: a bar's rung outcomes are only known one horizon later, so using them would be lookahead and would flatter the incumbent into an opponent the head could never fairly beat. Pass 3 walks oldest-to-newest, so "pushed more than horizon bars ago" is exactly "resolved by now". Each push is O(rungs), not O(window). The head's decision-rung Brier is pro-rated to the trailing estimate's coverage before the ratio, since the incumbent only scores bars where its window is warm. This line is worth reading on its own, independently of the head: if the trailing quantile beats the global constant, that is a cheap risk-control win available with no machine learning at all - and it is the same number either way, so the run answers both questions in one pass. The ring is deliberately NOT reset per era - it estimates the market, not the era, and re-warming 500 bars every era would leave the incumbent unusable over the first chunk of every scoring pass, handing the head a free win on exactly those bars. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:57:11 -04:00
//--- TRAILING-CLIMATOLOGY window, in bars, over RESOLVED outcomes only. This is the real incumbent for
//--- "replace a global ATR multiple": a rolling rung frequency adapts to the volatility regime, which is
//--- exactly the thing the head claims to predict, and it needs no model at all. ~2000 H1 bars is about
//--- three months - long enough that the rate is not noise, short enough to track a regime. The head's
//--- margin over THIS, not over a frozen constant, is what would justify 760 inputs.
#define EXCURSION_TRAIL_WINDOW 2000
//--- Resolved bars the trailing window must hold before its estimate is allowed to score anything.
#define EXCURSION_TRAIL_MIN_N 500
perf: the excursion head cost 3.6x era time - cut its dispatches ~250x Measured on exc-race-v3: LSTM era 300s -> 1087s (net 272->748s, "other" 30->337s). My estimate had been "single-digit percent". The cost is per-DISPATCH, not per-FLOP, and therefore hits EVERY backend: the head is 19k weights and ~2.4 GFLOP an era - seconds of arithmetic - but ~48k forward/backward calls x several layer submits each, and its 760-wide layer exceeds the CPU DLL's inline threshold so each one pays a real handoff. The classifier's own net time tripled too, from contention with a second pool on an already-full box. Three changes, all backend-neutral because they remove submits rather than tune threads: SCORE ONLY DISJOINT WINDOWS (~64x). Adjacent bars share all but one bar of their horizon, so 16k consecutive bars were always ~250 independent observations - the full-sample tally was never worth more than the disjoint one, it just quoted an n that was ~64x too large. Dropping it costs nothing statistically and removes 63 of every 64 forward passes. The two parallel tallies collapse into one, which is also less code. The trailing ring still advances on every bar: it needs the outcome SEQUENCE, and that is array lookups, not a forward pass. TRAIN ON EVERY 4th PRIMARY BAR (4x). The target is low-dimensional and strongly autocorrelated - neighbouring bars carry near-identical excursion information - so per-bar training buys resolution the target does not have. Strided on ATTEMPTS, not acceptances, so a stretch of unlabelled bars cannot silently change the spacing. OWN TIMING COLUMN. The head's passes were landing in the era line's "other" bucket, which is how a 3.6x regression read as an unexplained jump in the one column nobody attributes. A cost that cannot be seen in the timing line cannot be traded off against anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:38:29 -04:00
//--- Train the head on one primary bar in this many. Excursion targets are strongly autocorrelated -
//--- neighbouring bars share almost all of their horizon - so consecutive samples are near-duplicates,
//--- and the cost of this net is per-dispatch rather than per-FLOP on every backend.
#define EXCURSION_TRAIN_STRIDE 4
refactor(ai): derive the first dense layer's width instead of asking for it InitialNeurons was an input whose only defensible value depends on two things the user cannot see when picking from a dropdown: how wide the input vector ended up after feature selection, and how much in-sample data the study period actually yields. Left to a hand-picked constant it was badly wrong - 500 units against a 420-wide input is 210,500 weights, 72% of a 292,583-weight model, against ~36,500 training bars of which only ~2,236 are directional. That is 6.6 weights per training bar, and it EXPANDS a set of highly correlated inputs rather than compressing them. The symptom was already in the logs and had been read as a depth problem: the shallowest topology consistently beat the deepest (perceptron 52.7% balanced, hybrid 41.3%). Over-parameterization predicts that ordering just as well as covariate shift does, and only one of the two had been addressed. ComputeFirstLayerWidth() budgets roughly one first-layer weight per in-sample bar. Measured across the configurations in use: M15 10y -> 256 units, 129,071 weights, 0.73 per bar H1 10y -> 64 units, 28,727 weights, 0.65 per bar H4 10y -> 16 units, 7,559 weights, 0.68 per bar Two design points that matter: - It estimates in-sample bars from the STUDY PERIOD and timeframe, not from Bars(). What is downloaded grows over a terminal's lifetime, and a topology that widened as history filled in would re-key its own weights file and discard a trained model. - The result is snapped down to a coarse power-of-two ladder, so the estimate would have to be wrong by ~2x to change the answer. Every field it reads is already part of the weights-filename fingerprint, so the derived value needs no fingerprint entry of its own. The public setter is removed - it could only have been called after construction, and would either be ignored or silently re-key the model mid-run. Where the data cannot support even the floor (D1 over 10 years is under 2,000 bars) it now says so and names the fixes, rather than quietly training a model with more weights than examples. The DB config fingerprint drops the term too, which re-keys existing pattern databases once - correct, since a model an order of magnitude smaller should not inherit the old one's win-rate history. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 13:01:16 -04:00
#define FIRST_LAYER_MIN_WIDTH 16
refactor(ai): derive the dense taper's shape, not just its first layer Deriving the first layer's width left NeuronsReduction and MinNeuronsCount behind as inputs calibrated for something that no longer exists. Against a hand-picked 500-wide first layer "keep 30%, floor at 20" produced a genuine funnel - 500 -> 150 -> 45. Against the derived 64 it degenerates to 64 -> 20 -> 20: the reduction factor stops mattering after one step, and "minimum neurons per layer" silently becomes the width of every layer but the first. Two knobs whose labels no longer describe what they do. The taper now runs geometrically from the derived first-layer width down to a final hidden layer sized off the output count, spread evenly over however many layers the chosen AIType implies: MLP_3L 64 -> 28 -> 12 -> 3 29,151 dense weights MLP_4L 64 -> 37 -> 21 -> 12 -> 3 30,450 CONV/LSTM/HYBRID_2L 64 -> 12 -> 3 27,763 and it stays a funnel at the floor, where the old rule could not: D1 (first layer floored to 16) 16 -> 14 -> 12 -> 3 Both inputs are removed. With the width derived there is no freedom left in the taper, so keeping either would only let the user contradict the derivation. The layer COUNT stays selectable, because it is bundled into AIType alongside the conv/LSTM front-end - depth is an architecture choice, not a data-derived quantity, and pairing them means the two cannot contradict each other. m_minNeuronsCount / m_neuronsReduction survive as frozen members: nothing reads them to build a topology any more, but they hold positional slots in the .cfg sidecar and the weights fingerprint, and changing either value would re-key every model on disk for no behavioural reason. The DB config fingerprint drops both terms. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 14:03:42 -04:00
//--- Where the dense taper ENDS. The last hidden layer wants to be small enough to force the network to
//--- commit to a compressed representation, but comfortably wider than the decision itself so it is not
//--- the bottleneck - a few units per class is the usual heuristic. The absolute floor covers the
//--- single-output regression head, where 4x1 would be absurdly narrow.
feat(ai): real conv receptive field + the reference's channel pool CONV's convolution used window = step = one bar, which is a per-bar projection - a 1x1 conv with a temporal receptive field of ONE BAR. It never mixed information across time, so "convolutional" described the layer type and nothing about what it computed. Same finding that sank HYBRID's LSTM. Pooling was removed on 2026-07-29 for being misconfigured against the conv output's memory layout. That removal was right; leaving the conv at a one-bar window was not. The two belong together: the NeuroNet_DNG reference (references\MQL5\Experts\EDL\Trajectory.mqh layers 2-5, kernels byte-identical to ours) pairs conv(window=2, step=1, window_out=4) with pool(window=4, step=4), and the pool only earns its place because a conv with a real receptive field sits above it. The input is bar-major (BufferTempData appends m_neuronsCount contiguous features per bar), so a flat window of k*m_neuronsCount spans exactly k bars - the receptive field needed NO kernel change. The conv output is position-major, so window == step == window_out is a clean max-over-channels, which is what the reference does and what the existing pool kernels already implement correctly. New chain at H1 defaults (420 = 20 bars x 21): conv1 w=42 s=21 out=8 -> 19 pos x 8 = 152 pool w=8 s=8 -> 19 conv2 w=2 s=1 out=8 -> 18 pos x 8 = 144 (effective field: 3 bars) We deliberately stop before the reference's SECOND pool: a channel pool emits one scalar per position, so a trailing pool would hand the dense stack 18 values and force it to fan out 18 -> 64. That is a bottleneck below every learnable layer - the same class of mistake the 2026-07-29 removal was about. Fixes a latent sizing bug this exposed: CNet's conv/pool position cursor tracked sliding POSITIONS, but a conv's real width is units_count * window_out. Any pool stacked on a conv would therefore have sized against a width window_out times too small and silently built the wrong shape. Both branches now read the built layer's actual Neurons(), which is what the batch-norm branch already did for the same reason. Also closes the architecture-pinning trap: a .nnw persists the window each conv was built with, so an existing CONV/HYBRID model would have loaded cleanly and gone on training under the OLD architecture. The conv weight tensor is (window+1)*window_out, so this cannot be repaired in place - EnforceTopologyContract now detects it, reports both shapes, and retrains. Conv chain shape is derived in one place (ConvReceptiveFieldBars / ConvFirstStagePositions / HasSecondConvStage / ConvOutputPositions / ConvOutputWidth) and consumed by AddConvStage, LstmFanIn and the startup config line, so what is built and what is logged cannot drift. Both builds compile 0 errors, 0 warnings. Forces a CONV and HYBRID retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 20:05:37 -04:00
//--- Conv receptive field, in BARS. The reference (references\MQL5\Experts\EDL\Trajectory.mqh) uses a
revert(ai): restore the 4eae763 front-ends - both my rewrites stopped signaling CONV and LSTM were signaling at 4eae763. Two changes I made after it each broke one of them, and neither was caught by the metric I was reading. CONV, same data one hour apart on SP500 H1: 19:44 window = 1 bar era 5: dir-precision 18%, 45 live fires, best bal 3.1 -> 4.5 20:46 window = 2 bars era 5: no directional calls, 0 live fires, best bal frozen at era 1 LSTM: the sequence rewrite has been live since 18:48 (the `lstm 420->64` config line only derives that way under the new per-timestep budget) and has been OOS recall Neutral:100% with a flat IS error in every era since, past era 100. Both are reverted behind a switch rather than deleted, because both DIAGNOSES stand: a conv with a one-bar window is a 1x1 conv that cannot mix across time, and the old LSTM really did apply one gate step to the whole flattened input. What does not stand is shipping either on the strength of an offline correctness proof. CONV_RECEPTIVE_FIELD_BARS 2 -> 1 (also drops pool + conv2 via HasSecondConvStage, restoring the exact 4eae763 front-end) LSTM_SEQUENCE_MODE 0 (single-timestep layer; sizing follows, since a recurrence budgets on the per-step width and this one does not) Kept, because they are correct independent of the above: - per-layer dW/W era line (18f63c7) - the instrument that should have caught both of these in one era instead of a retrain cycle each - CNet conv/pool sizing-cursor fix (a pool stacked on a conv would have sized against a width window_out times too small) - .nnw architecture guard - forces the retrain this revert needs, since models trained since 20:46 carry a 2-bar conv tensor - LSTM_FORGET_BIAS_INIT and both offline checks (gradcheck 2.3e-10, flowcheck) - dormant at LSTM_SEQUENCE_MODE 0, correct when re-enabled The lesson is in the two #define comments: a gradient check proves the math, not that the layer trains in situ. Re-land each behind the dW/W line. Both builds compile 0 errors, 0 warnings. Forces a CONV/LSTM/HYBRID retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:04:23 -04:00
//--- window of 2 positions with step 1, stacked twice for an effective field of 3; a value of 2 here
//--- matches it at our bar granularity. Not an input: it is a structural property of the front-end, and
//--- a user who picks it is choosing against the pool and second-conv shapes derived around it.
//---
feat(ai): true multi-bar conv and true sequence LSTM CONV and LSTM were each configured as a strictly lossier perceptron, which is exactly what the panel showed: PAI 24% > CONV 18% > HYBRID 12% ~ LSTM 12%, monotone in how much reaches the dense stack (420 / 160 / 32 / 16). CONV - receptive field 1 -> 3 bars, and the pool is gone. Reading the reference kernels settled why 34d6aa4 killed CONV. FeedForwardConv emits POSITION-MAJOR output (matrix_o[out + window_out*i]), and FeedForwardProof is a flat contiguous max over `window` at stride `step`. On that layout any window <= window_out maxes ACROSS FILTERS within one position - it cannot pool over time at all. Our stage used window = step = filterCount: one max over all 8 filters per position, discarding 87.5% of the conv output and leaving only the argmax filter with gradient. That is a property of the reference's layout, not a porting bug, so there is no correct pool to swap in. Springenberg et al. ICLR 2015 is the answer already cited in this file: no pooling, get the hierarchy from strided convolution. The second conv went with it - its window was counted in raw elements while its comment claimed positions, so a "2-position" window actually spanned 2 filters of position 0. Filter count now derives from the WINDOW (RF * features / 2) instead of one bar, which at RF 3 was under-sizing the stage 3x. Shape: 20 bars x 21 -> 18 positions x 16 filters = 288. LSTM - sequence mode back on, forget bias 2.0 -> 1.0. The forward path rules out the "no gradient" reading of the 2026-07-30 failure: CPU_LSTMSeqForward starts every sample at h_0 = c_0 = 0 and unrolls that sample's own window, so nothing leaks between shuffled samples. Flat IS error + Neutral:100% is equally the signature of an output that does not vary with the input, and that is what bias 2.0 produces: c* = i*g/(1-sigmoid(b)) ~ 8.3*i*g, |c*| ~ 4.2, tanh pinned at 0.9995 with derivative 1e-3, so h_T is near-binary and set by the gate biases rather than the bars. Choosing 2.0 off the reach sweep was a method error - reach trades against saturation and the sweep never measured saturation. 1.0 is the Gers/Jozefowicz/Keras default and leaves tanh derivative ~0.1. Both builds 0/0. DLL unchanged (CPU_LSTMSeqForward/Backward already exported). Forces a retrain of CONV, LSTM and HYBRID - the .nnw pins architecture. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 17:42:33 -04:00
//--- RE-LANDED 2026-07-31 at 3, WITHOUT the channel pool that came with it the first time.
//--- Why the 34d6aa4 attempt failed, established by reading the reference kernels rather than guessing:
//--- CNeuronConvOCL emits POSITION-MAJOR output - `matrix_o[out + window_out * i]`, i.e.
//--- [pos0 f0..fN][pos1 f0..fN]... (References\MQL5\Experts\NeuroNet_DNG\NeuroNet.cl, FeedForwardConv).
//--- The reference pool (FeedForwardProof) is a flat contiguous max over `window` elements at stride
//--- `step`. On position-major data ANY window <= window_out therefore maxes ACROSS FILTERS INSIDE ONE
//--- POSITION - it cannot pool over time at all. The stage we built used window = step = filterCount,
//--- which is exactly one max over all 8 filters per position: 87.5% of the conv's output discarded, and
//--- only the argmax filter receiving gradient at each position. That is the measured regression (era 5:
//--- 18% dir-precision and 45 live fires at RF=1, versus "no directional calls" and 0 fires at RF=2).
//--- It is a property of the reference's own layout, not a porting mistake, so there is no "correct pool"
//--- to swap in here: pooling over time is not expressible on this layout without a transpose.
//--- The literature answer, already cited elsewhere in this file, is Springenberg et al. ICLR 2015
//--- ("Striving for Simplicity: The All Convolutional Net"): drop pooling, get the hierarchy from strided
//--- convolution instead. So this stage is now a single TRUE convolution - a CONV_RECEPTIVE_FIELD_BARS-bar
//--- window sliding one bar at a time - and nothing else. 3 bars is the smallest window that can express
//--- a turning point (before/at/after), which is what the ZigZag labels mark.
//--- Verify a change here against CNet::LayerLearningReport's per-era norm(d|W|%/|dW|%) line, not against
//--- accuracy: |dW| >> d|W| means the layer is rotating (learning), the two roughly equal with the norm
//--- falling means it is only being decayed away. Reading accuracy is what made the first regression take
//--- a full retrain cycle to spot.
#define CONV_RECEPTIVE_FIELD_BARS 3
revert(ai): restore the 4eae763 front-ends - both my rewrites stopped signaling CONV and LSTM were signaling at 4eae763. Two changes I made after it each broke one of them, and neither was caught by the metric I was reading. CONV, same data one hour apart on SP500 H1: 19:44 window = 1 bar era 5: dir-precision 18%, 45 live fires, best bal 3.1 -> 4.5 20:46 window = 2 bars era 5: no directional calls, 0 live fires, best bal frozen at era 1 LSTM: the sequence rewrite has been live since 18:48 (the `lstm 420->64` config line only derives that way under the new per-timestep budget) and has been OOS recall Neutral:100% with a flat IS error in every era since, past era 100. Both are reverted behind a switch rather than deleted, because both DIAGNOSES stand: a conv with a one-bar window is a 1x1 conv that cannot mix across time, and the old LSTM really did apply one gate step to the whole flattened input. What does not stand is shipping either on the strength of an offline correctness proof. CONV_RECEPTIVE_FIELD_BARS 2 -> 1 (also drops pool + conv2 via HasSecondConvStage, restoring the exact 4eae763 front-end) LSTM_SEQUENCE_MODE 0 (single-timestep layer; sizing follows, since a recurrence budgets on the per-step width and this one does not) Kept, because they are correct independent of the above: - per-layer dW/W era line (18f63c7) - the instrument that should have caught both of these in one era instead of a retrain cycle each - CNet conv/pool sizing-cursor fix (a pool stacked on a conv would have sized against a width window_out times too small) - .nnw architecture guard - forces the retrain this revert needs, since models trained since 20:46 carry a 2-bar conv tensor - LSTM_FORGET_BIAS_INIT and both offline checks (gradcheck 2.3e-10, flowcheck) - dormant at LSTM_SEQUENCE_MODE 0, correct when re-enabled The lesson is in the two #define comments: a gradient check proves the math, not that the layer trains in situ. Re-land each behind the dW/W line. Both builds compile 0 errors, 0 warnings. Forces a CONV/LSTM/HYBRID retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 10:04:23 -04:00
//--- Master switch for the sequence-LSTM front-end (AI\Network.mqh CNeuronLSTMOCL).
//--- 1 = the layer is a real recurrence over bars: one shared gate block applied at every timestep,
//--- with backpropagation-through-time. Gradient-checked to 2.3e-10 (DirectML\lstm_seq_gradcheck.cpp)
//--- and signal/gradient-reach-checked (DirectML\lstm_seq_flowcheck.cpp).
//--- 0 = the pre-2026-07-30 layer: ONE gate step over the whole flattened input, no recurrence.
//---
feat(ai): true multi-bar conv and true sequence LSTM CONV and LSTM were each configured as a strictly lossier perceptron, which is exactly what the panel showed: PAI 24% > CONV 18% > HYBRID 12% ~ LSTM 12%, monotone in how much reaches the dense stack (420 / 160 / 32 / 16). CONV - receptive field 1 -> 3 bars, and the pool is gone. Reading the reference kernels settled why 34d6aa4 killed CONV. FeedForwardConv emits POSITION-MAJOR output (matrix_o[out + window_out*i]), and FeedForwardProof is a flat contiguous max over `window` at stride `step`. On that layout any window <= window_out maxes ACROSS FILTERS within one position - it cannot pool over time at all. Our stage used window = step = filterCount: one max over all 8 filters per position, discarding 87.5% of the conv output and leaving only the argmax filter with gradient. That is a property of the reference's layout, not a porting bug, so there is no correct pool to swap in. Springenberg et al. ICLR 2015 is the answer already cited in this file: no pooling, get the hierarchy from strided convolution. The second conv went with it - its window was counted in raw elements while its comment claimed positions, so a "2-position" window actually spanned 2 filters of position 0. Filter count now derives from the WINDOW (RF * features / 2) instead of one bar, which at RF 3 was under-sizing the stage 3x. Shape: 20 bars x 21 -> 18 positions x 16 filters = 288. LSTM - sequence mode back on, forget bias 2.0 -> 1.0. The forward path rules out the "no gradient" reading of the 2026-07-30 failure: CPU_LSTMSeqForward starts every sample at h_0 = c_0 = 0 and unrolls that sample's own window, so nothing leaks between shuffled samples. Flat IS error + Neutral:100% is equally the signature of an output that does not vary with the input, and that is what bias 2.0 produces: c* = i*g/(1-sigmoid(b)) ~ 8.3*i*g, |c*| ~ 4.2, tanh pinned at 0.9995 with derivative 1e-3, so h_T is near-binary and set by the gate biases rather than the bars. Choosing 2.0 off the reach sweep was a method error - reach trades against saturation and the sweep never measured saturation. 1.0 is the Gers/Jozefowicz/Keras default and leaves tanh derivative ~0.1. Both builds 0/0. DLL unchanged (CPU_LSTMSeqForward/Backward already exported). Forces a retrain of CONV, LSTM and HYBRID - the .nnw pins architecture. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 17:42:33 -04:00
//--- RE-LANDED 2026-07-31, together with a corrected LSTM_FORGET_BIAS_INIT - see that constant, which is
//--- the part that was actually wrong.
//--- The 2026-07-30 attempt reported "flat IS error + Neutral:100%", which reads as "no gradient" but is
//--- equally the signature of an output that does not VARY with the input. The forward path rules the
//--- first out: CPU_LSTMSeqForward starts every sample at h_0 = c_0 = 0 (t==0 takes the nullptr branch),
//--- unrolls T = m_historyBars steps over that sample's own window, and emits h_T. State does not leak
//--- between shuffled samples, so shuffling is not the problem either. What DOES depend on the constant
//--- is saturation: with forget bias b the cell tends to c ~ i*g/(1-sigmoid(b)) over T steps, so b = 2.0
//--- gives c ~ 8*i*g, tanh(c) pins to +/-1, and h_T = o*tanh(c) becomes near-binary and set by the gate
//--- biases rather than by the bars. That is exactly "input-independent output".
//--- The tell in the era line is "OOS raw out B:min..max": a collapsed span there is this failure, not a
//--- gradient failure. Check it and the norm(d|W|%/|dW|%) line together before concluding anything.
#define LSTM_SEQUENCE_MODE 1
refactor(ai): derive the dense taper's shape, not just its first layer Deriving the first layer's width left NeuronsReduction and MinNeuronsCount behind as inputs calibrated for something that no longer exists. Against a hand-picked 500-wide first layer "keep 30%, floor at 20" produced a genuine funnel - 500 -> 150 -> 45. Against the derived 64 it degenerates to 64 -> 20 -> 20: the reduction factor stops mattering after one step, and "minimum neurons per layer" silently becomes the width of every layer but the first. Two knobs whose labels no longer describe what they do. The taper now runs geometrically from the derived first-layer width down to a final hidden layer sized off the output count, spread evenly over however many layers the chosen AIType implies: MLP_3L 64 -> 28 -> 12 -> 3 29,151 dense weights MLP_4L 64 -> 37 -> 21 -> 12 -> 3 30,450 CONV/LSTM/HYBRID_2L 64 -> 12 -> 3 27,763 and it stays a funnel at the floor, where the old rule could not: D1 (first layer floored to 16) 16 -> 14 -> 12 -> 3 Both inputs are removed. With the width derived there is no freedom left in the taper, so keeping either would only let the user contradict the derivation. The layer COUNT stays selectable, because it is bundled into AIType alongside the conv/LSTM front-end - depth is an architecture choice, not a data-derived quantity, and pairing them means the two cannot contradict each other. m_minNeuronsCount / m_neuronsReduction survive as frozen members: nothing reads them to build a topology any more, but they hold positional slots in the .cfg sidecar and the weights fingerprint, and changing either value would re-key every model on disk for no behavioural reason. The DB config fingerprint drops both terms. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 14:03:42 -04:00
#define HIDDEN_TAPER_OUTPUT_MULTIPLE 4
#define HIDDEN_TAPER_MIN_WIDTH 8
feat(nn): derive conv filter count and LSTM hidden size from the data Same defect the first-layer width had before 2026-07-29: both were inputs whose defaults were fixed constants picked with no reference to the input they sit on, which is the only thing that decides whether either number is sane. The conv layer is a per-bar projection - AddConvStage sets window = step = one bar's features - so its filter count should be read against the per-bar feature count. Sixteen filters COMPRESSED a 50-feature configuration 3x but EXPANDED a minimal 4-feature one 4x, and the expanding case adds parameters below every learnable layer without adding information. Now derived as half the per-bar feature count, snapped down a power-of-two ladder. The LSTM stage was the bigger miss. Its weight count is exactly 4*H*(H+inputs+1) (CNeuronLSTMOCL::SetInputs) and AddLstmStage feeds it the whole flattened vector, so the shipped 32 units against a 540-wide input is ~73k weights - more than DOUBLE the entire derived dense taper it feeds. It was the one stage the capacity budget never covered, which is why deriving the dense stack alone did not stop LSTM and HYBRID from being over-parameterized. Now solved from the same one-weight-per-in-sample-bar budget the first layer spends. Factored EstimatedInSampleBars() out of ComputeFirstLayerWidth so all three decisions spend one budget rather than each guessing at the training-set size separately. Both new values are assigned alongside the first-layer width, before the fingerprint that hashes them, and are functions of inputs already in that hash - so they need no entry of their own, and the same reasoning removes them from the DB config key. Both builds compile 0 errors, 0 warnings. Re-keys existing models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 09:22:11 -04:00
//--- ComputeConvFilterCount()/ComputeLstmHiddenSize() bounds. Both stages used to be inputs with a
//--- hand-picked constant default (16 filters, 32 hidden units) chosen with no reference to how wide the
//--- input actually ended up or how much data there is to fit them - the same defect the first-layer
//--- width had before it was derived. CONV_COMPRESSION_DIVISOR is the ratio the conv stage should
//--- compress a bar's feature vector by: the layer is a per-bar projection (window = step = one bar's
//--- features, see AddConvStage), so filters > features EXPANDS a correlated input at the very bottom of
//--- the stack, which is over-parameterization in its purest form. Halving is the conventional bottleneck
//--- choice and holds at any feature count.
feat(nn): derive dense depth, train on all history, pin the shape in .cfg Completes the derived-topology work. Three inputs removed. AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five entries instead of eight. Depth is now derived from the two endpoints the taper already has to connect (derived first-layer width, output-tied final width) at a 2x per-layer compression target, clamped [2..5]. Asking a user to pick a layer count while the code derives the widths those layers taper between was asking for half a decision: at 64 units tapering to 12, four layers compress by 1.4x per step and five by 1.3x, so the extra depth bought no abstraction. On the shipping H1/10y default the derivation lands on 3 layers - the depth that actually won Run 2. StudyPeriods removed. There is no case for training on less data than the broker provides at a ~6% directional base rate; the honest generalization read comes from the OOS holdout, not from withholding history. Training now starts at the earliest available bar, floored by MinTrainYear, which answers a different question (excluding dubious pre-history) and stays. That required closing the hazard the old code documented: the capacity budget now MEASURES the symbol's real bar count, and a topology derived from a measurement would widen as history downloads. Both ends are now pinned. Every derived value left the weights-filename fingerprint - keying a filename on a measured quantity means the EA looks for a file that does not exist, starts from era 0 and orphans a trained model, silently, because a missing cache is the normal first-run state. The shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the four derived fields rather than diffing them; a mismatch there would discard a fully-trained model over nothing the user did. Two fields appended to the .cfg for the conv/LSTM stages, length-guarded on read because FileReadInteger past EOF returns 0 with no error. ForceHiddenLayers, a compile-time constant like DebuggingMode, pins depth for diagnostic comparisons. It joins the fingerprint only when non-zero, so forced depths get their own files - sequential comparisons only, not simultaneous from one .ex5. Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64, 3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from ~58k to ~28k weights. Both builds compile 0 errors, 0 warnings. Re-keys existing models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
//--- ComputeHiddenLayerCount() bounds. HIDDEN_TAPER_TARGET_RATIO is the per-layer compression the taper
//--- aims for - halving, the conventional funnel - and depth is however many such steps it takes to get
//--- from the derived first-layer width to the output-tied final width. Two layers is the floor because
//--- one dense layer plus the head is a linear model with a single non-linearity; five is the ceiling
//--- because beyond it the per-step compression is so mild that the extra depth adds vanishing-gradient
//--- risk without adding abstraction.
#define HIDDEN_TAPER_TARGET_RATIO 2.0
#define MIN_HIDDEN_LAYERS 2
#define MAX_HIDDEN_LAYERS 5
//--- EstimatedInSampleBars() fallback for a chart whose history has not finished downloading. Below the
//--- trusted-bar floor the measurement says more about the sync state than about the symbol.
#define TOPOLOGY_BUDGET_MIN_TRUSTED_BARS 500
#define TOPOLOGY_BUDGET_FALLBACK_YEARS 10
feat(nn): derive conv filter count and LSTM hidden size from the data Same defect the first-layer width had before 2026-07-29: both were inputs whose defaults were fixed constants picked with no reference to the input they sit on, which is the only thing that decides whether either number is sane. The conv layer is a per-bar projection - AddConvStage sets window = step = one bar's features - so its filter count should be read against the per-bar feature count. Sixteen filters COMPRESSED a 50-feature configuration 3x but EXPANDED a minimal 4-feature one 4x, and the expanding case adds parameters below every learnable layer without adding information. Now derived as half the per-bar feature count, snapped down a power-of-two ladder. The LSTM stage was the bigger miss. Its weight count is exactly 4*H*(H+inputs+1) (CNeuronLSTMOCL::SetInputs) and AddLstmStage feeds it the whole flattened vector, so the shipped 32 units against a 540-wide input is ~73k weights - more than DOUBLE the entire derived dense taper it feeds. It was the one stage the capacity budget never covered, which is why deriving the dense stack alone did not stop LSTM and HYBRID from being over-parameterized. Now solved from the same one-weight-per-in-sample-bar budget the first layer spends. Factored EstimatedInSampleBars() out of ComputeFirstLayerWidth so all three decisions spend one budget rather than each guessing at the training-set size separately. Both new values are assigned alongside the first-layer width, before the fingerprint that hashes them, and are functions of inputs already in that hash - so they need no entry of their own, and the same reasoning removes them from the DB config key. Both builds compile 0 errors, 0 warnings. Re-keys existing models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 09:22:11 -04:00
#define CONV_COMPRESSION_DIVISOR 2
#define CONV_FILTERS_MIN 4
#define CONV_FILTERS_MAX 32
#define LSTM_HIDDEN_MIN 8
#define LSTM_HIDDEN_MAX 128
fix(training): escape the recall-gate catch-22 that let runs decay unchecked Evidence (MQL5\Logs, SP500 H1, 2026-07-29): Perceptron era 61 Buy 32% Sell 27% Neut 94% bal 51% LSTM era 160 Buy 16% Sell 11% Neut 98% bal 42% (peaked 49% @ era 44) Hybrid era 179 Buy 5% Sell 2% Neut 99% bal 35% (peaked 41%) CONV era 228 Buy 2% Sell 4% Neut 99% bal 35% (peaked 40% @ era 122) Every model peaks early then decays monotonically toward Neutral, and nothing stops it: the restore-best-weights + decay-eta handler is gated on m_bestPassedRecall, which stays false forever when no checkpoint ever clears the per-class floor. CONV ran 228 eras with eta pinned at its 0.000300 start. The plateau ladder cannot end such a run either (stage 3 refuses to deploy without a recall pass, so it resets ~27 times), making it a 1000-era one-way trip. The gate's own justification had expired. It was written when the pre-pass tiebreak was blended-accuracy-only, where "best" really did mean "called Neutral most confidently". The balanced-selection change replaced that with `balancedOosEra > m_bestBalancedOos` plus an isFullyCollapsedEra exclusion, so a Neutral-only era now scores ~33% - the FLOOR of the balanced metric - and cannot anchor the checkpoint at all. Pre-pass "best" now means "most class-balanced so far", which is worth defending; and isWorseEra is itself a balanced-accuracy regression, so it cannot fire merely for trading Neutral calls for Buy/Sell. The original concern still holds while the best-so-far IS near-collapse, so the escape is margin-guarded: defend the checkpoint only once balanced accuracy sits more than BALANCED_WORTH_DEFENDING_MARGIN_PCT (5pp) above the one-class floor of 100/3. Against the run above that engages for all three stuck topologies (42.3/41.3/50.0 vs a 38.3 threshold) while a genuinely collapsed run still explores freely. Two inputs restored to the regime that actually produced a deploy: - MinRecall 60 -> 40. The one successful auto-deploy in the logs (Hybrid, 28th 00:50, best balanced 66.0%) ran against a 40% floor. 60 has never been shown reachable here - a floor above what the config can reach is the same "target set too high" failure the surrounding comment already warns about. - OversampleParity 60 -> 90. 60 overcorrected. Runs now START Neutral-dominant (Buy 0-11% recall at era 1) and call Buy/Sell on 0-4% of bars against a ~6% true base rate - under-calling, with no headroom to converge down from. The deploying run began at Buy 90% / Sell 36%, 24% of bars called, and settled into the floor from above. Raw over-calling is the intended starting condition; live calls are base-rate-calibrated by AILogitPriorStrength, which is why the input's own note says to judge over-calling by live-fired precision, not raw counts. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 07:51:08 -04:00
#define BALANCED_COLLAPSE_PCT (100.0 / 3.0)
//--- How far above that floor a best-so-far checkpoint must sit before the regression handler is
//--- allowed to defend it during the pre-recall-pass phase. See the guard's comment in Train(): the
//--- point is to distinguish "the best we have is still basically a collapse, keep exploring freely"
//--- from "we found a real multi-class state and are now sliding off it", which is the case that ran
//--- unchecked for 228 eras on SP500 H1 (2026-07-29).
#define BALANCED_WORTH_DEFENDING_MARGIN_PCT 5.0
//--- PLATEAU LADDER ------------------------------------------------------------------------------
//--- The two branches above only fire on a NEW BEST (isBetterEra) or on a real REGRESSION
//--- (isWorseEra, a drop of more than ETA_DECAY_REGRESSION_PCT below the best). Between them sits a
//--- dead zone - "not better, not meaningfully worse" - where NOTHING happened: no checkpoint, no
//--- restore, no eta change. A run that settles into that band is stuck, and before this ladder
//--- existed it stayed stuck until the era cap (observed 2026-07-25: balanced accuracy pinned in a
//--- 64-69% band for 15 consecutive eras while eta sat frozen and the softmax outputs slowly
//--- compressed toward uniform, with ~970 eras still to burn before the cap would end it).
//---
//--- So: count eras since the last NEW BEST, and escalate when that count says the run has stopped
//--- improving on its own. Two ideas from the literature, in this order:
//--- 1. WARM RESTART (Loshchilov & Hutter, SGDR, ICLR 2017). On a plateau the correct move is a
//--- BIGGER step, not a smaller one - decaying eta into a plateau just entrenches whatever basin
//--- the model is sitting in. Note this is the opposite of the isWorseEra branch's decay, and
//--- deliberately so: decay answers overshoot, restart answers stagnation.
//--- 2. GAMMA ANNEALING (Mukhoti et al., "Calibrating Deep Neural Networks using Focal Loss",
//--- NeurIPS 2020, which schedules gamma DOWN over training rather than fixing it). Focal loss's
//--- (1-pt)^gamma modulator goes to ~0 on everything the model already classifies well, so late
//--- in a run the surviving gradient comes almost entirely from genuinely ambiguous bars - and
//--- near a pivot, ZigZag labels ARE ambiguous. Meanwhile WEIGHT_DECAY keeps pulling every
//--- weight toward zero on every step regardless (see AI\Network.mqh's note that a weight's
//--- sustainable magnitude is ~ its gradient SNR / WEIGHT_DECAY). Annealing gamma hands the
//--- easy-but-correct bars their gradient back, restoring the signal side of that ratio.
//--- Annealing is MONOTONE (gamma only ever decreases within a run), matching the scheduled-gamma
//--- literature; eta may still be bumped back up by the existing recovery bump.
//---
//--- Escalation is per-stage, every PLATEAU_PATIENCE_ERAS eras without a new best. ANY new best
//--- resets the counter and the stage to 0 (the ladder is a response to stagnation, so evidence the
//--- run is moving again retires it) - except the annealed gamma, which stays where it got to.
#define PLATEAU_PATIENCE_ERAS 8 // eras with no new best balanced accuracy before escalating a stage
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
#define PLATEAU_STAGE_RESTART 1 // first boosted warm restart (see PLATEAU_RESTART_BOOST)
#define PLATEAU_STAGE_ANNEAL 2 // second boosted warm restart (the gamma anneal it named is gone)
#define PLATEAU_STAGE_DEPLOY 3 // exhausted: deploy the best checkpoint and finish the run
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//--- Restart amplitude. Restoring eta merely TO its ceiling was a NO-OP whenever the run plateaued
//--- without ever tripping the regression decay - eta was still AT the ceiling, so stages 1 and 2
//--- assigned the value eta already had and the "ladder" was just a 24-era countdown to deploy
//--- (2026-08-09 audit, F2). Escaping a basin needs a rate LARGER than the one that settled into it:
//--- SGDR restarts span 10-100x; this is deliberately tamer because MAX_WEIGHT_DELTA and the
//--- best-checkpoint restore already bound the blast radius, and 5x the 3e-4 ceiling lands on 1.5e-3 -
//--- inside the ordinary Adam range (Kingma & Ba's own default is 1e-3). The boost is BOUNDED: the
//--- era-end anneal in Train() walks eta geometrically back to the ceiling over PLATEAU_PATIENCE_ERAS
//--- eras (a one-cycle kick, not a new permanent rate), and each restart also resets the optimizer's
//--- moment state (CNet::ResetOptimizerState) so the kick explores rather than replaying the stale
//--- momentum of the plateau it is escaping.
#define PLATEAU_RESTART_BOOST 5.0
refactor(ai): nine class-imbalance inputs down to two The imbalance section offered nine controls for one job. Audited against the code, five of them did not do what their names said at the shipped defaults: AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns whenever the adjusted loss is on, which is default. OversampleParity DEAD in training - Training.mqh gated the replay loop on !useLogitAdjustedLoss (correctly, citing Buda et al. 2018). Live only in the online-learning path. EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma damper - "replay minority bars through pass-2 oversampling" was a focal-loss switch. ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25. UseStaticPrior An exact duplicate of FreezePriorCalibration - the two were OR'd together in the single place either is read. So they were not five mechanisms fighting; they were one mechanism plus eight knobs that mostly described machinery that no longer ran. That is worse than a real conflict, because the log agreed with the names: the label-cache line printed "reps up to 28x (90% parity) (seeding era 0's class-balance oversampling)" on every run, describing an oversampling pass that had been switched off. It is fixed here too - it cost this session a wrong diagnosis. The one genuine redundancy was focal loss, running at gamma*0.125 alongside the adjusted loss: two corrections on the same axis, the exact stacking failure this file already cited Buda et al. for in two other places, damped by a replay flag whose replay path was itself dead. Removed rather than re-tuned. The plateau ladder is unaffected - its escape is the learning-rate warm restart; the gamma anneal beside it only ever stepped toward zero. WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze: LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted- Loss boolean, since a strength dial where 0 already means off does not need an on/off switch beside it. FreezePriorCalibration unchanged. It is the only one of the six corrections with a consistency guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection already ranks on - so the loss and the deploy decision optimize one thing. The online continual-learning path keeps its own alpha-balanced focal weight, now as constants pinned to the removed inputs' shipped defaults, so its behaviour is unchanged. It legitimately needs its own correction: ApplyLogitAdjustment() only runs inside a training run, so a deployed model that was reloaded carries no logit offsets and would otherwise stream 31:1 data into itself uncorrected. The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a double fed to a %d conversion and had always emitted a literal 0; the |MR: segment is written as the constant its shipped defaults produced. Dropping either would have re-keyed every model and forced a from-scratch retrain of the one topology currently converged and trading. Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable "neutralized by prior correction" diagnostic. Both builds compile 0 errors, 0 warnings. No retrain forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- PLATEAU_GAMMA_STEP removed 2026-07-31 with focal loss - the ladder escape is the warm restart.
//--- FILE-COMPATIBILITY SHIM for the removed "Min OOS accuracy % (converge)" input (was MinWR, default
//--- PCT_80). Convergence is decided by the plateau ladder now, so the value has no behavioural effect
//--- anywhere - but it still occupies a slot in TWO persisted identities that must not shift:
//--- 1. the topology .cfg layout (SaveTopologyConfiguration/LoadAndCompareTopologyConfiguration), and
//--- 2. the weights-FILENAME fingerprint (BuildConfigFingerprint) - change the hash and every existing
//--- model silently becomes unreachable and retrains from era 0.
//--- So keep writing/hashing the old default, and stop COMPARING the .cfg field (see that function) so a
//--- model saved under ANY previous MinWR still loads. Models trained with the shipped default 80 keep
//--- their exact filename and resume normally; one trained under a non-default MinWR gets a new filename
//--- and retrains once, which is unavoidable when a fingerprint field stops being a variable.
#define LEGACY_CONVERGE_WR_SLOT 80
feat(nn): derive dense depth, train on all history, pin the shape in .cfg Completes the derived-topology work. Three inputs removed. AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five entries instead of eight. Depth is now derived from the two endpoints the taper already has to connect (derived first-layer width, output-tied final width) at a 2x per-layer compression target, clamped [2..5]. Asking a user to pick a layer count while the code derives the widths those layers taper between was asking for half a decision: at 64 units tapering to 12, four layers compress by 1.4x per step and five by 1.3x, so the extra depth bought no abstraction. On the shipping H1/10y default the derivation lands on 3 layers - the depth that actually won Run 2. StudyPeriods removed. There is no case for training on less data than the broker provides at a ~6% directional base rate; the honest generalization read comes from the OOS holdout, not from withholding history. Training now starts at the earliest available bar, floored by MinTrainYear, which answers a different question (excluding dubious pre-history) and stays. That required closing the hazard the old code documented: the capacity budget now MEASURES the symbol's real bar count, and a topology derived from a measurement would widen as history downloads. Both ends are now pinned. Every derived value left the weights-filename fingerprint - keying a filename on a measured quantity means the EA looks for a file that does not exist, starts from era 0 and orphans a trained model, silently, because a missing cache is the normal first-run state. The shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the four derived fields rather than diffing them; a mismatch there would discard a fully-trained model over nothing the user did. Two fields appended to the .cfg for the conv/LSTM stages, length-guarded on read because FileReadInteger past EOF returns 0 with no error. ForceHiddenLayers, a compile-time constant like DebuggingMode, pins depth for diagnostic comparisons. It joins the fingerprint only when non-zero, so forced depths get their own files - sequential comparisons only, not simultaneous from one .ex5. Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64, 3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from ~58k to ~28k weights. Both builds compile 0 errors, 0 warnings. Re-keys existing models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
//--- Same treatment for the retired StudyPeriods input (removed 2026-07-30 - training now covers all
//--- available history). Its .cfg slot is positional and cannot be deleted without invalidating every
//--- deployed file, so a constant goes in and the field is no longer compared on load.
#define LEGACY_STUDY_PERIOD_SLOT 0
//--- LEGACY SLOT (was m_historyBars, DERIVED since 2026-08-11 - see DeriveHistoryBars). The literal
//--- is the shipped ind_Periods default, so every model trained at it keeps its filename; the real
//--- window lives in the .cfg (adopt-don't-compare, like every other derived shape field). A user
//--- who ran a non-default ind_Periods re-keys once, correctly - their old window is gone.
#define LEGACY_HISTORY_BARS_SLOT 20
//--- Derived-window rule: median confirmed swing leg, snapped DOWN to the ladder, capped. The floor
//--- is the ADZigZag Depth the feature comment always required; the cap is a wall-clock judgment -
//--- era time scales ~linearly with the window on every topology, and with the lag profile at the
//--- noise floor there is no evidence to buy more than ~1.6x today's cost on. Fallback = the old
//--- default, used when history is too thin to trust the measurement (same contract and warning as
//--- TOPOLOGY_BUDGET_MIN_TRUSTED_BARS - a window pinned from a handful of bars lasts a model's life).
#define HISTORY_BARS_FALLBACK 20
#define HISTORY_BARS_FLOOR 12
#define WINDOW_DERIVE_SPAN_BARS 20000
#define WINDOW_DERIVE_MIN_LEGS 30
#define WINDOW_SWING_WING 12
//--- Minimum true OOS samples a class needs this era before its recall is trusted as a real pass -
//--- see directionalRecallOK's declaration comment for the era-44-46 false-convergence this prevents.
//--- 10 is a low bar (still lets a genuinely thin early-run OOS window fall back to "not blocking"
//--- via recallPct==-1), just enough to rule out the zero/near-zero-sample degenerate case.
#define MIN_OOS_CLASS_SAMPLES_FOR_GATE 10
fix: the recall gate was unsatisfiable and the LR decay was a spiral Both made the run structurally unable to succeed, independently of any signal in the data. Found by reading the 13:01 log. RECALL GATE. m_objectiveMet required Buy, Sell AND Neutral OOS recall each >= 40%. First-touch resolution (ce52654) collapsed Neutral from the ~94% majority it was under exact-pivot labels to a same-bar-tie residue - 250 of 38,261 bars, 0.65% - so the floor was asking the model to identify 40% of coin-flip ties before it could converge. Measured: CONV, LSTM and HYBRID all logged "Neutral:0% (need >=40% each)" on every era. No model could ever satisfy it; every run was destined for the plateau ladder or the era cap. Only the DIRECTIONAL floors are load-bearing for the anti-collapse job the gate exists to do: an all-Neutral model shows Buy and Sell recall at 0% and is blocked by them. Neutral's own floor guarded the mirror bias (over-calling Buy/Sell at Neutral's expense), which was real at 94% prevalence and is not at 0.65% - there, almost never calling Neutral is correct rather than biased. Prevalence-guarded rather than hardcoded off, so it returns by itself if a future label rule makes Neutral substantial again. Deliberately NOT extended to Buy/Sell: exempting a thin directional class reopens the era-44-46 hole, which directionalRecallMeasured only half-covers - it checks those classes were MEASURED, not that they passed. ETA DECAY. A regressing era restored the checkpoint, reset the optimizer and cut eta - all on the FIRST regression. The next era then started from an identical state with a smaller step, regressed again, and got the same treatment. The loop is self-sustaining and cannot discover anything, because rolling the weights back is exactly what removes the exploration that would end it. Measured on PAI: eras 2-11 every one a regression against era 1, eta 0.000594 -> 0.000024, dW/W 0.000%/0.000% from era 2 onward. Ten eras, ~45s each, reproducing era 1 exactly and unable to do anything else. Now requires ETA_DECAY_PATIENCE_ERAS consecutive regressions - the standard ReduceLROnPlateau formulation. A single bad era is noise, and an improving era clears the counter so alternating runs never accumulate into a decay. Build tag -> gate-patience-v3. It had not moved in six commits, which is why the running binary could not be identified from its own log. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:28:58 -04:00
//--- A class this rare cannot be asked to carry the per-class recall floor. Applied to NEUTRAL ONLY -
//--- see the gate itself for why the directional classes are deliberately NOT exempted by prevalence.
//--- First-touch resolution (ce52654) collapsed Neutral from the ~94% majority it was under exact-pivot
//--- labels to a same-bar-tie residue: 250 of 38,261 bars, 0.65%. Demanding >=40% recall on that asks
//--- the model to spend capacity identifying coin-flip ties, and it is unreachable - measured
//--- 2026-08-10, CONV/LSTM/HYBRID all reported "Neutral:0% (need >=40% each)" every era, so no model
//--- could ever satisfy m_objectiveMet and every run was structurally unable to converge.
#define MIN_GATE_CLASS_SHARE_PCT 5.0
//--- Consecutive regressing eras before the checkpoint is restored and eta decayed. Was effectively 1:
//--- every regression rolled the weights back to the same checkpoint, reset the optimizer AND shrank
//--- eta, so the next era restarted from an identical state with a smaller step and regressed again -
//--- a geometric collapse with no exploration between rungs. Measured 2026-08-10 on PAI: eras 2-11 all
//--- regressed against era 1's best, eta fell 0.000594 -> 0.000024, and dW/W read 0.000%/0.000% from
//--- era 2 onward - ten eras that reproduced era 1 exactly and could not have done anything else.
//--- Patience is the standard ReduceLROnPlateau formulation and restores the exploration the rollback
//--- was removing: regress a few eras from the restored point BEFORE concluding the step is too big.
#define ETA_DECAY_PATIENCE_ERAS 3
//--- Bound on FindConfirmedZigZagPivot()'s backward scan (m_useSwingContext feature block) - ADZigZag's
//--- own stock defaults (Depth=12) produce pivots frequently enough in normal conditions that this cap
//--- should rarely bind, but a long, unusually strong single-direction run could genuinely go this long
//--- without a confirmed opposite-type pivot. Generous rather than tight: the scan is cheap (plain array
//--- reads, no indicator recompute) and its result is cached per-bar by BufferTempData()'s feature cache,
//--- so the one-time cost of a long scan is paid at most once per unique bar, not per training pass.
#define SWING_SCAN_CAP_BARS 750
//--- EMA shadow-weight deployment blend rate - see m_shadowNet's declaration comment for the full
//--- rationale. 0.01 matches the Tau range (0.001-0.01) used for target-network soft updates in
//--- Dmitriy Gizlyk's reference NeuroNet_DNG-based RL algorithms (references\MQL5\Experts\*\Study.mq5) -
//--- small enough that no single era's raw weights can move the deployed model far, large enough that
//--- the shadow still tracks real, sustained learning within a few dozen eras rather than lagging
//--- forever.
#define SHADOW_WEIGHT_TAU 0.01
feat: mini-batch gradient accumulation (F4), front-end-aware capacity budget (F6), split Wyckoff categoricals (N1) Completes the 2026-08-09 training audit. FORCES A RETRAIN of every Wyckoff-enabled config (N1 re-keys the fingerprint), and BOTH DLLs must be redeployed alongside the .ex5 - they carry new exports. F4 - mini-batch accumulation, TRAIN_BATCH_SIZE=32. Training was pure online SGD (one weight update per bar), which is the mechanical source of the era-to-era whipsaw every downstream guard was built to cope with. The O(n^2) outer product is native - AccumulateWeightGrad / AccumulateWeightGradConv / AccumulateBufferInto in Network.cl, WarriorCPU and WarriorDML - while the optimizer step is host-side MQL5 shared by all tiers (ApplyAccumToBlock), so there is one Adam/SGD implementation instead of four that can drift. - the LSTM needs no outer-product kernel (WeightsGradient already holds the sample's full dW) but could NOT simply be left un-zeroed between samples: CPU_LSTMSeqBackward/DML_LSTMSeqBackward memset it on entry. Hence a separate accumulator plus an elementwise add. - batch-norm gamma/beta accumulate in host arrays, not new BatchOptions slots - BN_OPT_STRIDE is baked into every persisted .nnw. - scoped to pass 2; online learning keeps immediate updates. Every save / checkpoint / scoring boundary flushes, scaling by the real sample count. - degrades to per-sample updates (one log line) on a tier that cannot accumulate, so old devices and DLL-free builds are unaffected. - verified offline: DirectML/batch_accum_check.cpp drives the real exports against an independent reference; at B=1 the accumulator matches the shipped unbatched kernel's own gradient to 1.1e-16. Math only - the in-situ check remains the per-layer dW/W report on a real era. F6 - ComputeFirstLayerWidth budgeted against the RAW input width even where a conv/LSTM front end had already reduced it, so an LSTM's dense stack was charged for 1,280 inputs when it receives 64. Confirmed from the deployed .cfg files: CONV, LSTM and HYBRID were all pinned at the 16-unit floor. Now budgeted against the front-end output and capped at it (never fan out), with the derivation reordered so both stages settle first. N1 - EventCode/EventPhase/StructuralPhase are signed categoricals packing direction and Wyckoff stage into one scalar across a sign discontinuity. Split into direction + [0,1] magnitude, the same convention the base OHLC block uses. Information-preserving; 13 readings now occupy 16 inputs. Compiled clean (0 errors, 0 warnings); both DLLs rebuilt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 11:48:03 -04:00
//--- MINI-BATCH SIZE (2026-08-09 audit, F4). Number of samples whose gradients are summed before ONE
//--- optimizer step is taken. 1 restores the exact per-sample behaviour this engine had until now.
//---
//--- Why it existed as a problem: training was pure online SGD - one weight update per bar - so the
//--- gradient driving each update was a single noisy sample and the end-of-era weight state was a
//--- high-variance draw. That is the mechanical source of the era-to-era whipsaw the plateau ladder,
//--- the checkpoint restore and the shadow EMA were all built to cope with downstream. Update noise
//--- falls as ~1/sqrt(B), so 32 cuts it by ~5.7x while taking 32x fewer (much better-estimated) steps.
//---
fix: the Adam second moment was never Adam - all four tiers Root cause of the B=32 regression, and it predates F4 entirely. Every Adam kernel stored v already square-rooted and then fed that stored value back in as if it were the variance: v_new = sqrt(b2 * v_old + (1 - b2) * g^2) That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below unit scale, so the denominator stops tracking the gradient and Adam degrades into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll (batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where a scale-invariant optimizer gives the same distance for both. After the fix all six magnitudes read 1.199 and v tracks |g| exactly. It hit conv/LSTM specifically because they sit behind a batch-norm with running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep in the degraded regime - while the dense stack near the loss stayed in the working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/ 0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already squared v back for gamma/beta and its comment named the kernels as wrong, which is exactly why gamma/beta kept training while the stages behind froze. Persisted .nnw needs no migration - v keeps its std-dev meaning. Also, the two ways F4 exposed it, both mine: - No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods (Krizhevsky 2014; Granziol et al. 2022), applied once in InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD. - Plateau patience denominated in eras, so raising B made the ladder 32x more impatient in its only unit. PAI converged at era 41 on ~49k updates where the same config had been finding new bests at era 1028. TrainPlateauPatienceEras() stretches it by the same sqrt(B). TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23 eras per stage, not 8 -> 45). Both helpers are identities at B=1. Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into the ranking key rather than checked at deploy time so a one-sided era cannot become best-so-far in the first place. Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded CAppDialog teardown that sat ahead of it - the same ordering inversion the rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit (vs ~1.1 s for the three that finished) having reached none of its cleanup, so its arrows stayed on the chart. Steps are now timed in the log. PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now walks every object type and reports the object counts when both are zero. Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt. FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
//--- REVISED 2026-08-09 (32 -> 8) after the first run at 32 regressed every topology. Two lessons,
//--- both measured, both now handled here rather than left implicit:
//---
//--- 1. An era is one PASS OVER THE DATA, and everything downstream that counts progress - the
//--- plateau ladder's patience, the selection metric, the eta anneal - is denominated in eras.
//--- Raising B does not change how much data an era sees; it divides how many optimizer STEPS
//--- that era takes. At 32 the perceptron declared convergence at era 41 on ~49k updates, where
//--- the same config had still been finding new bests at era 1028. The ladder had not become
//--- wrong, it had become 32x more impatient in the only unit it can measure.
//--- 2. Fewer steps must be paid for with a larger step. For SGD the compensation is linear in B
//--- (Goyal et al. 2017); for ADAPTIVE methods it is sqrt(B) - Krizhevsky 2014, derived for Adam
//--- specifically in Granziol et al. 2022. TrainBatchLrScale() below applies it.
//---
//--- 8 rather than 32 because the two costs compound: after the sqrt(B) LR bump an era still makes
//--- sqrt(B) less progress than the per-sample path, so PLATEAU_PATIENCE_ERAS has to stretch by the
//--- same sqrt(B) to stay equivalent (see TrainPlateauPatienceEras). At 32 that is 8 -> 45 eras per
//--- stage on a topology already taking 70 s/era; at 8 it is 8 -> 23, which is affordable. Noise
//--- still falls as 1/sqrt(B), so 8 keeps ~2.8x of the variance reduction that was the point.
//---
//--- 1 restores the exact per-sample behaviour this engine had until now: no accumulation, no LR
//--- scaling, no patience stretch (every helper below is an identity at B=1).
feat: mini-batch gradient accumulation (F4), front-end-aware capacity budget (F6), split Wyckoff categoricals (N1) Completes the 2026-08-09 training audit. FORCES A RETRAIN of every Wyckoff-enabled config (N1 re-keys the fingerprint), and BOTH DLLs must be redeployed alongside the .ex5 - they carry new exports. F4 - mini-batch accumulation, TRAIN_BATCH_SIZE=32. Training was pure online SGD (one weight update per bar), which is the mechanical source of the era-to-era whipsaw every downstream guard was built to cope with. The O(n^2) outer product is native - AccumulateWeightGrad / AccumulateWeightGradConv / AccumulateBufferInto in Network.cl, WarriorCPU and WarriorDML - while the optimizer step is host-side MQL5 shared by all tiers (ApplyAccumToBlock), so there is one Adam/SGD implementation instead of four that can drift. - the LSTM needs no outer-product kernel (WeightsGradient already holds the sample's full dW) but could NOT simply be left un-zeroed between samples: CPU_LSTMSeqBackward/DML_LSTMSeqBackward memset it on entry. Hence a separate accumulator plus an elementwise add. - batch-norm gamma/beta accumulate in host arrays, not new BatchOptions slots - BN_OPT_STRIDE is baked into every persisted .nnw. - scoped to pass 2; online learning keeps immediate updates. Every save / checkpoint / scoring boundary flushes, scaling by the real sample count. - degrades to per-sample updates (one log line) on a tier that cannot accumulate, so old devices and DLL-free builds are unaffected. - verified offline: DirectML/batch_accum_check.cpp drives the real exports against an independent reference; at B=1 the accumulator matches the shipped unbatched kernel's own gradient to 1.1e-16. Math only - the in-situ check remains the per-layer dW/W report on a real era. F6 - ComputeFirstLayerWidth budgeted against the RAW input width even where a conv/LSTM front end had already reduced it, so an LSTM's dense stack was charged for 1,280 inputs when it receives 64. Confirmed from the deployed .cfg files: CONV, LSTM and HYBRID were all pinned at the 16-unit floor. Now budgeted against the front-end output and capped at it (never fan out), with the derivation reordered so both stages settle first. N1 - EventCode/EventPhase/StructuralPhase are signed categoricals packing direction and Wyckoff stage into one scalar across a sign discontinuity. Split into direction + [0,1] magnitude, the same convention the base OHLC block uses. Information-preserving; 13 readings now occupy 16 inputs. Compiled clean (0 errors, 0 warnings); both DLLs rebuilt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 11:48:03 -04:00
//---
//--- Compile-time, not an input: it changes training dynamics but not the trained model's SHAPE, so it
//--- has no business in the weights fingerprint, and a user who picks a batch size is tuning something
//--- they cannot measure from the panel. Not every tier can honour it - see CNet::BatchSize.
fix: the Adam second moment was never Adam - all four tiers Root cause of the B=32 regression, and it predates F4 entirely. Every Adam kernel stored v already square-rooted and then fed that stored value back in as if it were the variance: v_new = sqrt(b2 * v_old + (1 - b2) * g^2) That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below unit scale, so the denominator stops tracking the gradient and Adam degrades into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll (batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where a scale-invariant optimizer gives the same distance for both. After the fix all six magnitudes read 1.199 and v tracks |g| exactly. It hit conv/LSTM specifically because they sit behind a batch-norm with running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep in the degraded regime - while the dense stack near the loss stayed in the working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/ 0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already squared v back for gamma/beta and its comment named the kernels as wrong, which is exactly why gamma/beta kept training while the stages behind froze. Persisted .nnw needs no migration - v keeps its std-dev meaning. Also, the two ways F4 exposed it, both mine: - No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods (Krizhevsky 2014; Granziol et al. 2022), applied once in InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD. - Plateau patience denominated in eras, so raising B made the ladder 32x more impatient in its only unit. PAI converged at era 41 on ~49k updates where the same config had been finding new bests at era 1028. TrainPlateauPatienceEras() stretches it by the same sqrt(B). TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23 eras per stage, not 8 -> 45). Both helpers are identities at B=1. Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into the ranking key rather than checked at deploy time so a one-sided era cannot become best-so-far in the first place. Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded CAppDialog teardown that sat ahead of it - the same ordering inversion the rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit (vs ~1.1 s for the three that finished) having reached none of its cleanup, so its arrows stayed on the chart. Steps are now timed in the log. PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now walks every object type and reports the object counts when both are zero. Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt. FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
#define TRAIN_BATCH_SIZE 8
//--- sqrt(B) learning-rate compensation and the matching patience stretch. Both are exactly 1.0 at
//--- B=1, so the whole mini-batch apparatus vanishes when TRAIN_BATCH_SIZE is 1.
//---
//--- Note this is only sound because the Adam second moment was fixed on 2026-08-09 (see
//--- AI\Network.cl's UpdateWeightsAdam): the previous recursion pinned its denominator at ~b2 for any
//--- gradient below unit scale, which made the step size proportional to |g| instead of invariant to
//--- it. Under THAT optimizer, shrinking the gradient by averaging cost a further sqrt(B) on top of
//--- the B fewer steps, and no learning-rate rule stated in terms of B could have compensated.
double TrainBatchLrScale(void) { return MathSqrt((double)TRAIN_BATCH_SIZE); }
int TrainPlateauPatienceEras(void) { return (int)MathRound(PLATEAU_PATIENCE_ERAS * MathSqrt((double)TRAIN_BATCH_SIZE)); }
//--- Online continual-learning tunables (see OnlineLearnStep()). Live-chart-only: once a model is
//--- deployed (m_trainingComplete) it keeps adapting to newly-RESOLVED bars as their barrier outcome
//--- becomes known - the same supervised triple-barrier task it was trained on, never trade P&L
//--- (that would be RL).
//--- ONLINE_LEARN_MAX_CATCHUP caps how many newly-confirmed bars one step may backprop, so a weekend
//--- gap or a restart can't stall a tick with an unbounded catch-up loop (excess is picked up over the
//--- next few bars). ONLINE_ACC_SMOOTH is the guardrail EMA horizon (bars) for the rolling predict-
//--- before-learn accuracy - far shorter than CNet::recentAverageSmoothingFactor (10000) so the
//--- guardrail actually reacts within a realistic live sample count. The deployed shadow is only blended
//--- toward Net while that rolling accuracy holds at/above max(ONLINE_LEARN_MIN_ACC, deploy_baseline -
//--- ONLINE_LEARN_ACC_MARGIN); if it drops, Net keeps adapting (so it can recover) but the blend is
//--- FROZEN so drift can never reach live - the conservative "reject the deployment of a bad update"
//--- guardrail (no weight-snapshot/revert needed). Persist every ONLINE_LEARN_PERSIST_EVERY updates so
//--- a crash loses at most that many bars of adaptation.
//---
//--- CLASS IMBALANCE. This path streams the RAW live class distribution one bar at a time, so calling
//--- backProp() at the default sampleWeight of 1.0 makes it a majority-class drift vector by
//--- construction - it would pull a balanced, converged model back toward Neutral. The streaming-safe
//--- correction is COST-level: alpha-balanced focal loss (Lin et al. 2017, "Focal Loss for Dense Object
//--- Detection", eq. 5), weight = alpha_c * (1 - p_t)^gamma, a SINGLE loss designed for exactly this
//--- ratio. alpha_c is inverse class frequency from the persisted priors (m_priorBuy/Sell/Neutral),
//--- majority normalised to 1.0, scaled by ONLINE_LEARN_PARITY and capped; gamma is
//--- ONLINE_LEARN_FOCAL_GAMMA. ONLINE_LEARN_MAX_CLASS_WEIGHT is the uncapped ceiling,
//--- ONLINE_LEARN_ALPHA_CAP the tighter one actually applied.
//--- The two engines deliberately use DIFFERENT corrections. Train() corrects analytically in the
//--- gradient via the logit-adjusted loss, which is NOT available here: ApplyLogitAdjustment() only
//--- runs inside a training run, so a deployed-then-reloaded model carries no offsets and would
//--- otherwise stream skewed data in with no correction at all. These were shared inputs until
//--- 2026-07-31 and are now constants at those inputs' shipped defaults - behaviour is unchanged.
//--- ONLINE_LEARN_ETA_SCALE: this path previously inherited whatever value the GLOBAL `eta` happened to
//--- be left at by the last Train() chunk of ANY model instance (PAI/CONV/LSTM/HYBRID share it), which
//--- is arbitrary. eta is now pinned to this model's own converged rate scaled down for the duration of
//--- the loop, then restored, so a live adaptation step is deliberately gentler than a training step.
#define ONLINE_LEARN_MAX_CATCHUP 64
#define ONLINE_ACC_SMOOTH 50.0
#define ONLINE_LEARN_WARMUP 20
#define ONLINE_LEARN_MIN_ACC 40.0
#define ONLINE_LEARN_ACC_MARGIN 10.0
#define ONLINE_LEARN_PERSIST_EVERY 32
#define ONLINE_LEARN_MAX_CLASS_WEIGHT 5.0
refactor(ai): nine class-imbalance inputs down to two The imbalance section offered nine controls for one job. Audited against the code, five of them did not do what their names said at the shipped defaults: AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns whenever the adjusted loss is on, which is default. OversampleParity DEAD in training - Training.mqh gated the replay loop on !useLogitAdjustedLoss (correctly, citing Buda et al. 2018). Live only in the online-learning path. EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma damper - "replay minority bars through pass-2 oversampling" was a focal-loss switch. ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25. UseStaticPrior An exact duplicate of FreezePriorCalibration - the two were OR'd together in the single place either is read. So they were not five mechanisms fighting; they were one mechanism plus eight knobs that mostly described machinery that no longer ran. That is worse than a real conflict, because the log agreed with the names: the label-cache line printed "reps up to 28x (90% parity) (seeding era 0's class-balance oversampling)" on every run, describing an oversampling pass that had been switched off. It is fixed here too - it cost this session a wrong diagnosis. The one genuine redundancy was focal loss, running at gamma*0.125 alongside the adjusted loss: two corrections on the same axis, the exact stacking failure this file already cited Buda et al. for in two other places, damped by a replay flag whose replay path was itself dead. Removed rather than re-tuned. The plateau ladder is unaffected - its escape is the learning-rate warm restart; the gamma anneal beside it only ever stepped toward zero. WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze: LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted- Loss boolean, since a strength dial where 0 already means off does not need an on/off switch beside it. FreezePriorCalibration unchanged. It is the only one of the six corrections with a consistency guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection already ranks on - so the loss and the deploy decision optimize one thing. The online continual-learning path keeps its own alpha-balanced focal weight, now as constants pinned to the removed inputs' shipped defaults, so its behaviour is unchanged. It legitimately needs its own correction: ApplyLogitAdjustment() only runs inside a training run, so a deployed model that was reloaded carries no logit offsets and would otherwise stream 31:1 data into itself uncorrected. The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a double fed to a %d conversion and had always emitted a literal 0; the |MR: segment is written as the constant its shipped defaults produced. Dropping either would have re-keyed every model and forced a from-scratch retrain of the one topology currently converged and trading. Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable "neutralized by prior correction" diagnostic. Both builds compile 0 errors, 0 warnings. No retrain forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- Pinned to the shipped defaults of the removed OversampleParity (90%), ConstrainReplay (true ->
//--- cap 3.0) and FocalLossGamma (1.0) inputs - see the CLASS IMBALANCE note above.
#define ONLINE_LEARN_PARITY 0.9
#define ONLINE_LEARN_ALPHA_CAP 3.0
#define ONLINE_LEARN_FOCAL_GAMMA 1.0
#define ONLINE_LEARN_ETA_SCALE 0.25
//--- Free function, not a class method: needed by CExpertSignalAIBase's constructor init list for
//--- both m_modelEta and m_etaCeiling (see their declaration comments), which runs before member-init
//--- order could safely make one depend on another - this only touches the TrainingOptimizer input
//--- (Variables\Inputs.mqh) and the SgdLearningRate/AdamLearningRate inputs (AI\Network.mqh's `lr`
//--- macro resolves to AdamLearningRate), never class members.
fix: the Adam second moment was never Adam - all four tiers Root cause of the B=32 regression, and it predates F4 entirely. Every Adam kernel stored v already square-rooted and then fed that stored value back in as if it were the variance: v_new = sqrt(b2 * v_old + (1 - b2) * g^2) That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below unit scale, so the denominator stops tracking the gradient and Adam degrades into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll (batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where a scale-invariant optimizer gives the same distance for both. After the fix all six magnitudes read 1.199 and v tracks |g| exactly. It hit conv/LSTM specifically because they sit behind a batch-norm with running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep in the degraded regime - while the dense stack near the loss stayed in the working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/ 0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already squared v back for gamma/beta and its comment named the kernels as wrong, which is exactly why gamma/beta kept training while the stages behind froze. Persisted .nnw needs no migration - v keeps its std-dev meaning. Also, the two ways F4 exposed it, both mine: - No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods (Krizhevsky 2014; Granziol et al. 2022), applied once in InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD. - Plateau patience denominated in eras, so raising B made the ladder 32x more impatient in its only unit. PAI converged at era 41 on ~49k updates where the same config had been finding new bests at era 1028. TrainPlateauPatienceEras() stretches it by the same sqrt(B). TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23 eras per stage, not 8 -> 45). Both helpers are identities at B=1. Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into the ranking key rather than checked at deploy time so a one-sided era cannot become best-so-far in the first place. Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded CAppDialog teardown that sat ahead of it - the same ordering inversion the rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit (vs ~1.1 s for the three that finished) having reached none of its cleanup, so its arrows stayed on the chart. Steps are now timed in the log. PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now walks every object type and reports the object counts when both are zero. Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt. FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
//--- The sqrt(B) mini-batch compensation is applied HERE, at the one point that decides the base rate,
//--- so it reaches m_etaCeiling, m_modelEta, the plateau ladder's boost and its anneal from a single
//--- edit rather than being re-derived at each of them. Identity at TRAIN_BATCH_SIZE 1.
double InitialEtaForOptimizer(void)
{
fix: the Adam second moment was never Adam - all four tiers Root cause of the B=32 regression, and it predates F4 entirely. Every Adam kernel stored v already square-rooted and then fed that stored value back in as if it were the variance: v_new = sqrt(b2 * v_old + (1 - b2) * g^2) That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below unit scale, so the denominator stops tracking the gradient and Adam degrades into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll (batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where a scale-invariant optimizer gives the same distance for both. After the fix all six magnitudes read 1.199 and v tracks |g| exactly. It hit conv/LSTM specifically because they sit behind a batch-norm with running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep in the degraded regime - while the dense stack near the loss stayed in the working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/ 0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already squared v back for gamma/beta and its comment named the kernels as wrong, which is exactly why gamma/beta kept training while the stages behind froze. Persisted .nnw needs no migration - v keeps its std-dev meaning. Also, the two ways F4 exposed it, both mine: - No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods (Krizhevsky 2014; Granziol et al. 2022), applied once in InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD. - Plateau patience denominated in eras, so raising B made the ladder 32x more impatient in its only unit. PAI converged at era 41 on ~49k updates where the same config had been finding new bests at era 1028. TrainPlateauPatienceEras() stretches it by the same sqrt(B). TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23 eras per stage, not 8 -> 45). Both helpers are identities at B=1. Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into the ranking key rather than checked at deploy time so a one-sided era cannot become best-so-far in the first place. Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded CAppDialog teardown that sat ahead of it - the same ordering inversion the rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit (vs ~1.1 s for the three that finished) having reached none of its cleanup, so its arrows stayed on the chart. Steps are now timed in the log. PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now walks every object type and reports the object counts when both are zero. Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt. FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
return ((TrainingOptimizer == SGD) ? SgdLearningRate : lr) * TrainBatchLrScale();
}
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//+------------------------------------------------------------------+
//| Uniform random index in [0, n) for Fisher-Yates shuffles. |
//| MQL5's MathRand() is 15-bit (0..32767). `MathRand() % n` with |
//| n > 32768 makes the modulo the IDENTITY on the generator's range: |
//| a swap target above 32767 can never be drawn, so any shuffle over |
//| more elements than that - every full-history H1 training queue - |
//| was provably non-uniform, partially re-admitting the correlated |
//| same-class gradient runs the shuffle exists to break (2026-08-09 |
//| audit, F1; see AI\Network.mqh's MAX_WEIGHT_DELTA comment for why |
//| those runs matter). Two draws give 30 uniform bits; the residual |
//| modulo bias at n ~ 1e5 against 2^30 is ~1e-4 - negligible here. |
//+------------------------------------------------------------------+
int ShuffleRandomIndex(const int n)
{
if(n <= 1)
return 0;
int r30 = ((MathRand() & 0x7FFF) << 15) | (MathRand() & 0x7FFF);
return r30 % n;
}
class CExpertSignalAIBase : public CExpertSignalCustom
{
protected:
string ID;
//--- ID with the bracketed config tag stripped - "Hybrid 3L [HYB-9369]" -> "Hybrid 3L". The tag is a
//--- topology prefix plus a fingerprint hash: it exists to tell one CHART's model files from another's
//--- when reading the journal or the State\ folders, which is a developer's problem, not an owner's.
//--- Logs and the VerboseMode panels keep the full ID; the plain-language panels use this. Strips from
//--- the LAST " [" so a model name containing a bracket could not truncate the whole label.
string DisplayName(void) const
{
int cut = StringFind(ID, " [");
int next = cut;
while(next >= 0)
{
cut = next;
next = StringFind(ID, " [", cut + 1);
}
return (cut >= 0 ? StringSubstr(ID, 0, cut) : ID);
}
CiOpen m_Open;
CiClose m_Close;
CiHigh m_High;
CiLow m_Low;
CiVolumes m_Volumes;
CiTime m_Time;
//--- optional classic-indicator input features (m_useMA/m_useRSI/m_useMACD/m_useIchimoku below) -
//--- independent instances from the ones Signals\SignalMA.mqh/SignalRSI.mqh/SignalMACD.mqh/
//--- SignalIchimoku.mqh use for trade voting, since this class and those CSignal* classes are
//--- unrelated hierarchies (feature engineering vs. signal vote). The MA
//--- feature now uses the SAME unified indicator as the classic vote (CustomIndicators\ADMovingAverage.mq5)
//--- via CiCustom, so its type/period are tunable (see ADIndicatorTuner maType/maPeriod).
CiCustom m_MA;
CiRSI m_RSI;
//--- built-in Ci* wrappers, not CiCustom: MACD is defined on plain EMAs and Ichimoku on plain
//--- highest/lowest midpoints, so there is no unified AD* custom indicator to route them through the
//--- way m_MA goes through ADMovingAverage. Periods come from the tuner (macdFast/macdSlow/
//--- macdSignal, ichiTenkan/ichiKijun/ichiSenkou).
CiMACD m_MACDFeature;
CiIchimoku m_Ichimoku;
//--- custom price-action/volume indicators (CustomIndicators\*.mq5), loaded via iCustom/CiCustom
CiCustom m_ADCumulativeDelta;
CiCustom m_ADShorteningOfThrust;
CiCustom m_ADWyckoffEventStream;
CiCustom m_ADWyckoffFailedStructure;
CiCustom m_ADWyckoffSignificantBarInversion;
fix: the trailing incumbent read the future across eras; cold AD blocks cached zeros as truth Three findings from the 2026-08-11 audit: 1. The excursion head's trailing-quantile ring was deliberately never cleared between eras ("a rolling estimate of the market, not of the era") - but pass 3 re-walks the SAME OOS window every era, so at each walk's restart the ring still held the outcome masks of the newest OOS bars from the previous walk: the chronological FUTURE of the bars about to be scored. For the first ~window+horizon pushes of every era the "trailing" incumbent was partly a leading one - conservative for the gate (an informed incumbent is a harder hurdle) but exactly the self-made-artifact class 06d4785 hunts. The ring now clears at era-score reset; the warm-up bars simply don't score the trail race, which the m_excTrailN gating already accounts for. 2. skillTrail compared the head's FULL-block Brier (pro-rated by coverage) against the incumbent's subset sum - valid only if head skill is uniform across the OOS walk, while the trail-scored subset systematically excludes each era's warm-up bars. The audit also found m_excBrierHeadD/BaseD/ m_excOosHitsD declared, zeroed and never accumulated (dead since e2c9593 made every scored bar disjoint). The dead trio is replaced by m_excBrierHeadT: the head's Brier accumulated only on the bars the warm incumbent also scored, so the race now compares both predictors on an identical bar set. 3. The AD/Wyckoff feature blocks read GetData with no EMPTY_VALUE guard; a cold (still-calculating) indicator returns EMPTY_VALUE everywhere, the sanitize loop rewrote that to 0.0, and the bar SUCCEEDED - so BufferTempData cached an all-zero Wyckoff block as a success for the whole bar frame: the one path the f6150ee only-cache-successes rule cannot see, because it never fails (the ba13eef class, arriving through values that never fail; a resumed model's era-0 prebuild starts milliseconds after OnInit). ADIndicatorCold() probes the NEWEST bar - EMPTY_VALUE there means async warm-up (transient reject, retried), while deep bars beyond the buffered depth keep the sanitize loop's neutral-fill so degraded history still trains. Also fixed m_featureCacheValid's declaration comment, which still described the pre-f6150ee cached-miss semantics. Compile: 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 18:20:24 -04:00
//--- "is this AD indicator still calculating?" probe - see the definition in Features.mqh for
//--- why cold must be a TRANSIENT rejection and not a zero-fill (2026-08-11)
bool ADIndicatorCold(CiCustom &ind);
2026-08-13 10:23:11 -04:00
//--- GetTickCount() stamp of the last pass-1 sweep in which EVERY window failed on a transient
//--- cause (cold indicator). Non-zero arms a short era-start backoff so the retry loop stops
//--- starving the very indicator threads it is waiting on - see Train()'s fresh-era block.
uint m_coldSweepTick;
//--- ground truth for the training labels (see m_swingConfirmationBars' declaration comment) -
//--- CustomIndicators\ADZigZag.mq5, a renamed/rebranded copy of the stock MQL5 ZigZag indicator,
//--- always created (not gated behind an Enable* input like the AD* feature indicators above,
//--- since it isn't an optional input feature - it IS the label) and always run at its own stock
//--- defaults (Depth=12, Deviation=5, Backstep=3) - never touched by AutoTuneIndicators, since
//--- tuning the ground truth itself alongside the model being scored against it would let a trial
//--- "improve" its OOS score by cherry-picking an easier target rather than a better model.
CiCustom m_ADZigZag;
//--- currently-active tunable input values for each AD indicator (AutoTuneIndicators search space)
//--- plus their own flatten/unflatten/perturb/best-tracking logic - see CADIndicatorTuner's
//--- declaration comment (Expert\ADIndicatorTuner.mqh) for why this is a separate collaborator
//--- rather than loose fields/methods here: it's entirely self-contained (never touches Net,
//--- Train()'s state machine, or anything else in this class), unlike TuneIndicatorsAndTrain()
//--- below, which orchestrates Train()/Net/checkpointing around it and stays here for exactly that
//--- reason. InpContextMode/InpSessionType/InpSessionCount are session choices, not accuracy
//--- knobs, and are hardcoded to the indicators' own defaults in InitAD*() rather than stored/
//--- tuned in the collaborator.
CADIndicatorTuner m_indicatorTuner;
bool m_autoTuneIndicators;
//--- rebuilds only the AD* indicator handles (in place) so ReInit picks up updated param structs
bool ReInitADIndicators(CIndicators *indicators);
2026-08-13 10:23:11 -04:00
//--- installs a loaded/saved param set into the tuner and rebuilds the handles ONLY when the set
//--- actually differs from what the live indicators already run - see the definition for the
//--- resume-time churn this exists to avoid.
bool AdoptIndicatorParams(const double &loaded[], CIndicators *indicators);
//--- builds a fresh, untrained topology into Net (assumes Net is currently NULL); factored out of
//--- InitNeuralNetwork() so TuneIndicatorsAndTrain() can rebuild weights per trial without
//--- re-running indicator init (which would re-Add() the base indicators into indicators a 2nd time)
bool BuildFreshTopology();
//--- CIndicators collection passed into InitNeuralNetwork(), retained so
//--- TuneIndicatorsAndTrain() can call ReInitADIndicators() between trials
CIndicators *m_indicatorsPtr;
CNet *Net;
//--- EMA "shadow" copy of Net, blended a small step (SHADOW_WEIGHT_TAU) toward Net's weights at
//--- the end of every era (see Train()'s era-end block) rather than replaced outright. Live
//--- trading/inference (RefreshLatestSignal()) reads from this, not from Net directly, so any
//--- single era's raw weights - including an Adam overshoot event - can only ever nudge what's
//--- actually deployed by SHADOW_WEIGHT_TAU, not overwrite it wholesale. Bootstrapped as a clone of
//--- Net (via the same Save()/Load() pattern m_simOosNet uses) the first time InitNeuralNetwork()
//--- runs with no prior shadow file, and persisted alongside Net.Save() thereafter. NULL only
//--- during the brief window before that first bootstrap completes - every read site falls back to
//--- Net in that case, never blocks on it.
CNet *m_shadowNet;
//--- One-shot latch for EnsureShadowNet()'s clone bootstrap. Cloning a second full net can fail on the
//--- tester's CPU-DLL compute fallback (the chart's GPU/DirectML path succeeds); without this the
//--- bootstrap would retry on EVERY bar, re-initialising the compute backend each time - log spam and a
//--- multi-second-per-bar crawl through an inference-only backtest. A failed/skipped bootstrap is
//--- harmless: every read site falls back to Net, and a bootstrapped shadow is identical to Net until
//--- era-end blending diverges it (which never happens in a no-training run). Reset with m_shadowNet.
bool m_shadowBootstrapAttempted;
//--- Online continual-learning state (see OnlineLearnStep(); tunables at ONLINE_LEARN_* above). Live-
//--- chart-only - a deployed (m_trainingComplete) model keeps learning from newly-confirmed ZigZag
//--- structure as bars mature, waiting the full m_swingConfirmationBars confirmation delay just like
//--- training so a still-provisional (repainting) recent bar is NEVER backpropped. m_enableOnlineLearning
//--- is the input gate. m_onlineLearnedUpToTime is the bar-TIME watermark of the newest bar already
//--- learned from (indexed by time, not now-relative index, so it survives the per-bar index-frame
//--- shift); persisted in the .stats sidecar (WST3). m_onlineRollingAcc is the guardrail EMA (0-100)
//--- of predict-before-learn hits (seeded from the deploy OOS baseline), m_onlineSamples the cumulative
//--- update count (both persisted). m_onlineBarsSincePersist drives periodic saves (in-memory only).
bool m_enableOnlineLearning;
datetime m_onlineLearnedUpToTime;
double m_onlineRollingAcc;
long m_onlineSamples;
int m_onlineBarsSincePersist;
//--- Latched log state so the guardrail freeze/resume transition prints once per flip, not per bar.
bool m_onlineBlendFrozen;
CArrayDouble *TempData;
double dError;
double dUndefine;
double dForecast;
double dPrevSignal;
//--- ALTERNATION GATE REMOVED 2026-08-01 with the triple-barrier relabel. m_lastNonNeutralSignal
//--- suppressed any live Buy following another Buy with no Sell between. That was CORRECT under the
//--- exact-pivot target (a ZigZag can only emit a new pivot by FLIPPING type, so a repeat was
//--- provably a false fire) and the premise died with the target: a triple-barrier label is answered
//--- independently at every bar, so ten consecutive Buy setups inside one trend are simply correct.
//--- Do not reinstate it. It also meant a one-sided (`Sell:0%`) model got ONE trade per backtest,
//--- because the awaited opposite signal that reopens the gate never came.
diag: inference-path census, to explain zero-trade backtests A backtest of the CONVERGED CONV model produced "Final directional result: 0.00000000" on every one of 1744 bars and therefore zero trades. Nothing in the log could separate the three candidate causes, and each needs a different fix: 1. RefreshLatestSignal never called (new-bar gate never fires) 2. called, but bailing at one of its two early returns 3. running fine, and the model genuinely answers Neutral every bar Counts all three plus the Buy/Sell/Neutral split, printed once at shutdown via StopTraining (which the tester reaches through OnDeinit). Three increments per bar against a full feedForward - not worth gating. Ruled out while writing this, so the next session does not re-derive it: - the alternation gate (m_lastNonNeutralSignal) is NOT the cause. It starts at Neutral, so a first Buy would still fire and show up as one non-zero direction. We saw zero. It IS still a live hazard for a one-sided model - CONV currently calls Buy:17% Sell:0%, and after the first Buy every later Buy is suppressed until a Sell that never comes - but it cannot explain an all-zero run. - shallow buffers do not hard-fail the feature builder: the swing-context Donchian loop breaks gracefully when it runs off loaded history. It does mean converged-path inference computes Donchian/return/SMA features over a TRUNCATED window versus training, which is a real train/inference skew worth its own fix, but it degrades features rather than zeroing them. Both builds 0/0. Diagnostic only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 18:24:32 -04:00
//--- Inference-path census counters - see PrintInferenceTally() for why these exist. Cheap enough to
//--- keep unconditionally: three increments per bar against a full feedForward.
long m_refreshOk;
long m_refreshFailFeatures;
long m_refreshFailShort;
long m_refreshBuy;
long m_refreshSell;
long m_refreshNeutral;
//--- VOTE-GATE census. RefreshLatestSignal() can answer Buy on hundreds of bars while
//--- LongCondition()/ShortCondition() still return 0 on every one, because those open with a
//--- readiness gate the refresh path never consults:
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- if(!m_trainingComplete && !(m_inferenceOnly && m_modelLoadedFromDisk)) return 0;
//--- In the tester that reduces to "the seeded _optcache.nnw must have LOADED"; when it has not, the
//--- run silently produces direction 0.00 on every bar. Without these two the census reads as "the
//--- model answers Neutral" - false, and it points at a completely different fix. They separate the
//--- model's ANSWER from whether that answer was allowed to become a vote.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
long m_voteGateBlocked; // directional decisions the readiness gate discarded
long m_voteGatePassed; // directional decisions that became a real vote
//--- Flag pair as of the first vote attempt, latched so the tally can name WHICH half of the gate
//--- failed rather than just reporting that it did. -1 = no vote was ever attempted.
int m_voteGateCompleteAtFirst;
int m_voteGateLoadedAtFirst;
//--- m_gateSnapshot removed with the alternation gate above - it existed only to roll that gate back
//--- when the parent discarded this filter's vote. The AI signal now holds no one-shot vote state, so
//--- BeginVote()/RevokeVote() fall back to the base class's empty implementations.
//--- Non-max suppression (NMS) of directional signals. Consecutive H4 bars near a real turn share
//--- almost their entire feature vector, so a single reversal fires the same class on a whole run of
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- adjacent bars - a visual/emission cluster around one event, not several distinct turns.
//--- DEFAULT OFF since 2026-08-01: suppressing repeats was justified by the exact-pivot target, and
//--- under triple-barrier labels a run of same-direction setups is real signal, so collapsing it
//--- DISCARDS trades rather than de-duplicating them. Kept opt-in for one marker per move.
//--- Keeps only the FIRST (earliest) bar of a same-direction run, dropping same-direction neighbours
//--- within m_signalClusterWindow bars; 0 disables. Per-direction, so a missed opposite signal never
//--- blocks a later reversal (unlike the alternation gate), and causal, so the live signal and the
//--- drawn history declutter identically. Post-processing ON TOP of the model - the raw per-bar
//--- recall/precision/accuracy stats stay un-NMS'd so they keep measuring the network itself.
//--- The cluster extends from the last SEEN same-direction bar, not the last KEPT one: measuring
//--- from the kept bar re-emitted an arrow every window+1 bars inside a long run.
int m_signalClusterWindow;
//--- Per-bar predicted signed signal for THIS era (index = now-relative bar index; -2 = not scored
//--- this era). When NMS is on, the scan passes (1/2/3) ONLY record into this cache and draw nothing;
//--- PruneDirectionalClusters() then renders the whole declustered set once at era end. That is what
//--- keeps the chart from ever showing the raw mid-era clusters (the passes would otherwise draw every
//--- above-threshold bar as it scanned, and the sweep only cleaned up at era end). Recording (not
//--- drawing) also decouples NMS from pass 2's SHUFFLED order, which no inline cursor could dedup.
//--- Sized to bar count each fresh era.
double m_arrowSignalCache[];
//--- Live-side NMS state: bar TIME of the last SEEN signal per direction (advances on every same-
//--- direction bar, kept or suppressed, so a contiguous live run collapses to one) plus the cached
//--- accept/suppress decision for that exact bar (keeps repeated same-bar RefreshLatestSignal calls
//--- idempotent - re-evaluating the same bar returns its first decision, not a flipped one). 0 = none.
datetime m_nmsLiveBuyTime;
datetime m_nmsLiveSellTime;
bool m_nmsLiveBuyAccept;
bool m_nmsLiveSellAccept;
//--- Last KEPT live signal of either direction, for cross-direction resolution: a Buy and a Sell
//--- within m_signalClusterWindow bars are flicker at one turn zone (real opposite pivots are a whole
//--- leg apart), so only the higher-confidence side is kept. Confidence = |signed signal| = winning
//--- softmax probability. Same rule runs in PruneDirectionalClusters() for the historical chart.
datetime m_nmsLiveKeptTime;
ENUM_SIGNAL m_nmsLiveKeptDir;
double m_nmsLiveKeptConf;
datetime dtStudied;
long m_eraCount; // cumulative era counter, persisted in the .nnw so restarts don't look like they reset progress
bool m_trainingComplete; // persisted: true only once Train() converged (objective+stability), not just interrupted
bool bEventStudy;
//--- out-of-sample holdout: share (%) of the study period never trained on, used only to
//--- measure genuine forward accuracy so overfitting shows up in the stats, not just live/OOS trading
int m_oosSplitPct;
double dOosError; // smoothed OOS mismatch rate (0..100), lower is better
double dOosForecast; // smoothed OOS accuracy (0..100)
int m_oosSamples; // count of OOS predictions evaluated this Train() call
//--- per-era raw (pre-softmax) output-neuron stats over pass 3's OOS scan, reset at pass 3
//--- start; surfaced in the era-end log line. The per-bar logit spread (max-min across the 3
//--- outputs) is the collapse fingerprint: a healthy net differentiates bars (avg spread well
//--- above 0), a saturated all-one-class net pins all three outputs to the same value on every
//--- bar (avg spread ~0, mins/maxes flat) - see CLASS_LOGIT_SCALE's comment (AI\Network.mqh).
double m_oosOutMin[3];
double m_oosOutMax[3];
double m_oosOutSpreadSum;
int m_oosOutCount;
//--- per-era counts of the network's own classification of each bar it fed forward (IS+OOS),
//--- reset at the start of every era; surfaced in the status label text so class imbalance
//--- (e.g. the network collapsing to all-Neutral) is visible while training runs
int m_countBuySignals;
int m_countSellSignals;
int m_countNeutralSignals;
//--- per-era counts of the *true* label of every bar fed forward (IS+OOS), reset alongside the
//--- predicted counts above. Used both to display the actual class distribution in the status
//--- label text and, more importantly, to drive IS-sample oversampling in Train() (see backProp calls
//--- below) - rarer classes get replayed more times so gradient descent doesn't just learn to
//--- always call the majority (usually Neutral) class.
int m_trueBuyCount;
int m_trueSellCount;
int m_trueNeutralCount;
//--- snapshot of the class totals above, taken at the end of the PREVIOUS era (see Train()'s
//--- era-reset block) and held fixed for the whole of the current era. The oversampling ratio
//--- below is computed from these frozen counts rather than the live, still-accumulating
//--- m_true*Count values - using the live counts made the ratio order-dependent within a single
//--- era (chronological, oldest-to-newest bar processing means whichever class happens to be
//--- numerically behind at any given moment gets amplified up to 5x, even if that's just a
//--- transient artifact of which regime the era's early bars came from, not the class's true
//--- overall rarity) - a real source of run-to-run oscillation in the predicted class mix. All
//--- zero on era 0, when oversampling simply falls back to 1x (see the reps calc in Train()).
int m_prevEraTrueBuyCount;
int m_prevEraTrueSellCount;
int m_prevEraTrueNeutralCount;
//--- per-era OOS confusion counts, reset each era; used to compute per-class OOS recall (hits/total)
//--- for the status label text and, more importantly, as an additional convergence gate alongside the
//--- blended dOosForecast accuracy - a model that "wins" only by calling everything Neutral will
//--- have high dOosForecast but near-zero Buy/Sell recall, and should NOT be allowed to converge.
int m_oosBuyHits, m_oosBuyTotal;
int m_oosSellHits, m_oosSellTotal;
int m_oosNeutralHits, m_oosNeutralTotal;
//--- same per-era OOS confusion counts as above but keyed by PREDICTED class instead of true class,
//--- i.e. per-class precision (of the bars this era where the model called Sell, how many actually
//--- were Sell?) rather than recall (of the bars that actually were Sell, how many did it catch?).
//--- Recall alone can't distinguish "the model over-fires Sell and happens to also catch enough real
//--- Buys/Neutrals to clear their recall floors" from "the model is genuinely well-calibrated" - a
//--- skewed Predicted-this-era count (see m_countSellSignals) with recall still passing the gate is
//--- exactly that failure mode, and precision is what would expose it.
int m_oosBuyPredicted, m_oosBuyPredictedHits;
int m_oosSellPredicted, m_oosSellPredictedHits;
int m_oosNeutralPredicted, m_oosNeutralPredictedHits;
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
//--- WOULD THE TRADE HAVE PAID. Same predicted-class keying as the *Hits pair above, but scored
//--- against m_winLongCache/m_winShortCache instead of against label agreement - "the Buy calls whose
//--- long actually reached target before stop", not "the Buy calls that matched a 3-class label".
//---
//--- These two questions coincided only while reward >= risk forced Buy and Sell to partition the
//--- resolved bars. Removing the minimum-reward:risk raise (2026-08-09) ended that: the measured
//--- geometry puts the target NEARER than the stop, so both directions can win on the same bar, the
//--- label can only name one of them, and the other call was being scored as an error despite its
//--- trade paying in full. The gap is not academic - it moved the zero-skill reference to 37.5% while
//--- break-even sat at 67.3%, i.e. the gate would happily ship a model that loses on every trade.
//---
//--- With this pair, dirPrecPct IS the win rate, and the invariant the gate rests on is restored
//--- exactly: an always-Buy model wins P(winLong) = m/(m+k) of the time, which is also the break-even
//--- rate for a k:m trade, so "beats chance" and "is profitable" are once again the same test.
int m_oosBuyPredictedWins;
int m_oosSellPredictedWins;
//--- Zero-skill denominators, MEASURED rather than assumed from m/(m+k): how many scored bars a long
//--- (resp. a short) would have won on, regardless of what the model called or what the label says.
//--- max() of the two is what an always-call-one-direction model scores, which is the reference the
//--- deploy gate needs. Measured, because the theoretical identity holds for a driftless walk and real
//--- instruments drift - SP500 makes always-long genuinely better than a coin, and a gate that used
//--- the textbook value would credit that drift to the model.
int m_oosWinLongTotal;
int m_oosWinShortTotal;
//--- Confidence calibration: the classification head's forward pass still uses SIGMOID per-neuron
//--- (see BuildFreshTopology()'s SIGMOID comment - bounded activation, avoids logit runaway), but
//--- the BACKWARD pass (CNet::backProp/backPropOCL in AI\Network.mqh) now trains all 3 neurons
//--- jointly against a true softmax+categorical-cross-entropy gradient (softmax_i - target_i), not
//--- 3 independent binary-cross-entropy targets. ApplyClassificationSoftmax() at READ time
//--- reproduces the same normalization the loss was actually trained against, so the value it
//--- returns is now a genuinely loss-consistent probability (still not literature-perfect
//--- calibration - no temperature scaling/Platt scaling has been applied - but no longer
//--- structurally decoupled from what was optimized). SignedAIConfidence()/g_AISignedConfidence
//--- exposes it, and Money\MoneyIntelligent.mqh:AdjustRiskAmount() scales position-sizing risk%
//--- directly off it.
//--- m_oosConfidenceSum accumulates MathAbs(dPrevSignal) (the claimed confidence) over every
//--- classified OOS bar this era; compared against this era's actual OOS accuracy
//--- ((m_oosBuyHits+m_oosSellHits+m_oosNeutralHits)/m_oosSamples) at era-end to derive
//--- m_confidenceCalScale - a single empirical multiplier ("the model claims 80% on average but is
//--- only right 60% of the time -> scale reported confidence by 0.75") applied to the MAGNITUDE
//--- only, never the sign/class decision, in SignedAIConfidence(). EMA-blended across eras (same
//--- smoothing factor as dOosForecast) so one noisy era can't swing it, and clamped to [0.3, 1.5]
//--- so a thin/degenerate OOS window can't drive it to something absurd. Starts at 1.0 (no
//--- correction) until the first era with OOS samples computes a real value.
double m_oosConfidenceSum;
double m_confidenceCalScale;
//--- minimum acceptable OOS recall (%) for the Buy and Sell classes individually before Train() is
//--- allowed to declare convergence; a class with zero OOS samples this era doesn't block (avoids a
//--- deadlock when a given era's OOS window happens to contain no examples of that class)
int m_minDirectionalRecallPct;
//--- There is deliberately NO minimum-confidence input here any more, and no member holding one.
//--- Confidence is expressed through the VOTE WEIGHT (ConfidenceTier() -> PatternWeightForTier()),
//--- not through a separate entry floor, so a weak call is not blocked at the AI's own boundary - it
//--- votes weakly and is then filtered by the same Min vote to open threshold that filters a weak
//--- classic vote. That is what makes ONE input genuinely govern both engines.
//--- The floor it replaced was a scale mismatch: Min_Vote_Open fed it directly as a 0..1 probability
//--- while ALSO being the 0..100 averaged-vote threshold, and a 3-class argmax winner is
//--- arithmetically >= 1/3, so every setting from 0 to 33 gated precisely nothing while every setting
//--- above that silently re-quartiled the confidence tiers as a side effect. See ConfidenceTier().
refactor(ai): nine class-imbalance inputs down to two The imbalance section offered nine controls for one job. Audited against the code, five of them did not do what their names said at the shipped defaults: AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns whenever the adjusted loss is on, which is default. OversampleParity DEAD in training - Training.mqh gated the replay loop on !useLogitAdjustedLoss (correctly, citing Buda et al. 2018). Live only in the online-learning path. EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma damper - "replay minority bars through pass-2 oversampling" was a focal-loss switch. ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25. UseStaticPrior An exact duplicate of FreezePriorCalibration - the two were OR'd together in the single place either is read. So they were not five mechanisms fighting; they were one mechanism plus eight knobs that mostly described machinery that no longer ran. That is worse than a real conflict, because the log agreed with the names: the label-cache line printed "reps up to 28x (90% parity) (seeding era 0's class-balance oversampling)" on every run, describing an oversampling pass that had been switched off. It is fixed here too - it cost this session a wrong diagnosis. The one genuine redundancy was focal loss, running at gamma*0.125 alongside the adjusted loss: two corrections on the same axis, the exact stacking failure this file already cited Buda et al. for in two other places, damped by a replay flag whose replay path was itself dead. Removed rather than re-tuned. The plateau ladder is unaffected - its escape is the learning-rate warm restart; the gamma anneal beside it only ever stepped toward zero. WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze: LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted- Loss boolean, since a strength dial where 0 already means off does not need an on/off switch beside it. FreezePriorCalibration unchanged. It is the only one of the six corrections with a consistency guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection already ranks on - so the loss and the deploy decision optimize one thing. The online continual-learning path keeps its own alpha-balanced focal weight, now as constants pinned to the removed inputs' shipped defaults, so its behaviour is unchanged. It legitimately needs its own correction: ApplyLogitAdjustment() only runs inside a training run, so a deployed model that was reloaded carries no logit offsets and would otherwise stream 31:1 data into itself uncorrected. The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a double fed to a %d conversion and had always emitted a literal 0; the |MR: segment is written as the constant its shipped defaults produced. Dropping either would have re-keyed every model and forced a from-scratch retrain of the one topology currently converged and trading. Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable "neutralized by prior correction" diagnostic. Both builds compile 0 errors, 0 warnings. No retrain forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- Stops the per-era EMA update of the measured class priors after the first real measurement, so
//--- the tau*log(prior) offsets stay pinned to the distribution the run started from.
bool m_freezePriorCalibration;
refactor(ai): nine class-imbalance inputs down to two The imbalance section offered nine controls for one job. Audited against the code, five of them did not do what their names said at the shipped defaults: AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns whenever the adjusted loss is on, which is default. OversampleParity DEAD in training - Training.mqh gated the replay loop on !useLogitAdjustedLoss (correctly, citing Buda et al. 2018). Live only in the online-learning path. EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma damper - "replay minority bars through pass-2 oversampling" was a focal-loss switch. ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25. UseStaticPrior An exact duplicate of FreezePriorCalibration - the two were OR'd together in the single place either is read. So they were not five mechanisms fighting; they were one mechanism plus eight knobs that mostly described machinery that no longer ran. That is worse than a real conflict, because the log agreed with the names: the label-cache line printed "reps up to 28x (90% parity) (seeding era 0's class-balance oversampling)" on every run, describing an oversampling pass that had been switched off. It is fixed here too - it cost this session a wrong diagnosis. The one genuine redundancy was focal loss, running at gamma*0.125 alongside the adjusted loss: two corrections on the same axis, the exact stacking failure this file already cited Buda et al. for in two other places, damped by a replay flag whose replay path was itself dead. Removed rather than re-tuned. The plateau ladder is unaffected - its escape is the learning-rate warm restart; the gamma anneal beside it only ever stepped toward zero. WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze: LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted- Loss boolean, since a strength dial where 0 already means off does not need an on/off switch beside it. FreezePriorCalibration unchanged. It is the only one of the six corrections with a consistency guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection already ranks on - so the loss and the deploy decision optimize one thing. The online continual-learning path keeps its own alpha-balanced focal weight, now as constants pinned to the removed inputs' shipped defaults, so its behaviour is unchanged. It legitimately needs its own correction: ApplyLogitAdjustment() only runs inside a training run, so a deployed model that was reloaded carries no logit offsets and would otherwise stream 31:1 data into itself uncorrected. The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a double fed to a %d conversion and had always emitted a literal 0; the |MR: segment is written as the constant its shipped defaults produced. Dropping either would have re-keyed every model and forced a from-scratch retrain of the one topology currently converged and trading. Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable "neutralized by prior correction" diagnostic. Both builds compile 0 errors, 0 warnings. No retrain forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- THE class-imbalance correction: tau in Menon et al. 2021's logit adjustment. tau*log(prior_c) is
//--- added to each class logit in the TRAINING gradient only, so the network absorbs the offset and
//--- its RAW argmax is already balanced-error-optimal at inference - no second correction at read
//--- time. 0 disables it. This is the sole survivor of the nine-input imbalance section audited away
//--- on 2026-07-31 (see Variables\Inputs.mqh); it is the only one of them with a consistency
//--- guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection
//--- already ranks on. m_logitAdjustLogged keeps the once-per-run tau/cap report to one line.
feat(ai): logit-adjusted loss, replacing oversampling and the post-hoc prior Menon et al. 2021 (ICLR), "Long-tail learning via logit adjustment": add tau*log(prior_c) to each class logit inside the training gradient. Softmax CE on adjusted logits is consistent for BALANCED error - the metric checkpoint selection already ranks on - so the loss and the deploy decision finally optimize the same thing. The engine already computed a true softmax + categorical-CE gradient and wrote it over the per-neuron sigmoid delta, so this is an offset added to three logits in the two places that gradient is built (backProp scalar path and backPropOCL). No backend, kernel or DLL change; the forward pass and every inference path are untouched, which is the point - the network learns to absorb the offset, so its raw argmax becomes the balanced-optimal decision with nothing applied at inference. Replaces rather than stacks. Minority replay is disabled while this is on, and the post-hoc inference prior is forced off. Stacking is not a theoretical worry: simulated on the measured 1118/1119/34298 distribution in the weak-signal regime, plain CE collapses to Neutral (33.4% balanced, Buy 0%), replay reaches 48.1%, logit adjustment 50.9% with better balance - and BOTH together score 45.4% with Neutral recall at 0%, worse than either alone. Buda et al. 2018 predicts exactly that. Motivation from the six-chart run: every topology took one direction to ~50% recall and abandoned the other, the direction chosen arbitrarily (the batch-norm control went Buy 1% / Sell 42%, the inverse of the other five). One era in 1,301 cleared the per-class recall floor. Fingerprinted conditionally, so the converged 60.7% models on disk keep their filenames and stay loadable as the fallback. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:05:14 -04:00
double m_logitAdjustTau;
fix(ai): cap logit-adjustment strength to the head's usable logit range tau=1.0 inverted the collapse instead of curing it. The head is SIGMOID, so each output is bounded to [0,1] and the widest logit gap the net can express between two classes is CLASS_LOGIT_SCALE * (1-0) = 6. The offsets are tau*log(prior_c), whose spread on this 30:1 imbalance is 3.42 - so tau=1.0 spent 57% of the ENTIRE expressible range on the prior correction. The network did the only thing available to it: saturate Buy/Sell outputs to 1.0 to overcome a -3.42 training handicap. The offsets are absent at inference, so that surplus made every bar directional. Measured across all five still-training charts: Neutral recall 0%, directional calls on ~100% of bars, win rate 5-7% against a ~6% base rate - no information whatsoever - while balanced accuracy read a flattering 58-64% because two of its three terms sat near 95%. OOS accuracy 6%. Menon et al. assume an unbounded logit head where a 3.42 shift is negligible against the reachable range. It is not negligible here, so the strength is now expressed RELATIVE to the range actually available: tau_eff = min(tau_cfg, LOGIT_ADJUST_MAX_RANGE_FRACTION * SCALE / spread) At 20% that gives tau 0.35 on this data. Deliberately a fraction rather than a tau ceiling: it stays correct if CLASS_LOGIT_SCALE changes, if the head becomes unbounded, or on any symbol whose imbalance differs. The input remains effective below the cap, so dialling it down needs no rebuild. Simulated at a signal strength where the task is genuinely learnable, the precision/recall frontier is monotone: tau 1.0 -> 49.6% call rate at 6.4% precision (base rate 6.1%, i.e. worthless); tau 0.35 -> 2.0% at 15.5%; tau 0.15 -> 0.2% at 33.3%. The capped value lands in the same regime the pre-logit-adjustment run occupied (1-6% of bars at 20-35% win rate). Also logs the measured priors, the spread, and whether the cap bound. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 23:20:07 -04:00
bool m_logitAdjustLogged;
fix: the imbalance correction never ran during the auto-tune search Neutral collapse on all four topologies by era 5 with a 2:6 barrier (recall Buy 0% / Sell 0% / Neutral 100%), and the panel stuck on "measuring...". One root cause, and it was not the barrier. The labels were fine: Buy 25.4% / Sell 22.0% / Neutral 52.5%, which is exactly gambler's ruin for m=2,k=6 (2/8 = 25% per side), with only 0.1% of Neutral coming from the vertical barrier - so the new m*k horizon scaling is right, arguably generous. What was broken: Train()'s era-start block wrapped UpdateClassPriors() in `if(!m_evalMode)`. The auto-tune GA scores every candidate in eval mode, and AutoTuneIndicators ships ON, so on a default configuration EVERY era of the search ran with unmeasured priors. ApplyLogitAdjustment() requires measured priors; without them it calls ClearLogitAdjustment() and returns. So the entire search trained under PLAIN cross-entropy. With a 52.5% majority class the optimum of plain CE is "always predict Neutral", and that is precisely what all four models found. The panel followed: its counters only advance on bars the model CALLED Buy or Sell, so a collapsed model leaves them at zero and the line reads "measuring..." forever. This was latent, not new. It has been true for every auto-tuned run, but it was invisible while the labels were near-balanced - last night's accidental 1:1 barrier gave 43/40/17, where plain CE has no majority to collapse into. Widening the stop to 2*ATR (correctly - 1*ATR is too tight to survive noise) moved Neutral to the majority and exposed it. The guard's stated fear cannot happen. These priors are measured from the LABEL distribution, and the tuner only perturbs indicator periods (MA/RSI/MACD/Ichimoku/AD). The barrier label depends on ATR, SL_Mode and TP_Mode - none of which the search touches - so every candidate sees byte-identical labels and identical priors. There is nothing to contaminate. What the guard actually protected was the .stats write, and that is gated separately: eval candidates never checkpoint and never persist. Also, because this is the THIRD quiet no-op to cost a run in this codebase (after the fictional oversampling log line and the shadow-blend skip): - ApplyLogitAdjustment() now WARNS when it declines to install, instead of silently clearing. A mechanism that cannot announce it is not running is indistinguishable from one that is. - The panel distinguishes "measuring..." (before era 1, nothing scored yet - an honest warm-up) from "no directional calls yet" (eras trained, zero calls - a finding, not a wait). Both builds compile 0 errors / 0 warnings. No retrain forced by this commit itself, but the collapsed models must be discarded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:46:24 -04:00
//--- Latch for the counterpart warning: the correction DECLINING to install. See ApplyLogitAdjustment().
bool m_logitAdjustSkipWarned;
feat(ai): logit-adjusted loss, replacing oversampling and the post-hoc prior Menon et al. 2021 (ICLR), "Long-tail learning via logit adjustment": add tau*log(prior_c) to each class logit inside the training gradient. Softmax CE on adjusted logits is consistent for BALANCED error - the metric checkpoint selection already ranks on - so the loss and the deploy decision finally optimize the same thing. The engine already computed a true softmax + categorical-CE gradient and wrote it over the per-neuron sigmoid delta, so this is an offset added to three logits in the two places that gradient is built (backProp scalar path and backPropOCL). No backend, kernel or DLL change; the forward pass and every inference path are untouched, which is the point - the network learns to absorb the offset, so its raw argmax becomes the balanced-optimal decision with nothing applied at inference. Replaces rather than stacks. Minority replay is disabled while this is on, and the post-hoc inference prior is forced off. Stacking is not a theoretical worry: simulated on the measured 1118/1119/34298 distribution in the weak-signal regime, plain CE collapses to Neutral (33.4% balanced, Buy 0%), replay reaches 48.1%, logit adjustment 50.9% with better balance - and BOTH together score 45.4% with Neutral recall at 0%, worse than either alone. Buda et al. 2018 predicts exactly that. Motivation from the six-chart run: every topology took one direction to ~50% recall and abandoned the other, the direction chosen arbitrarily (the batch-norm control went Buy 1% / Sell 42%, the inverse of the other five). One era in 1,301 cleared the per-class recall floor. Fingerprinted conditionally, so the converged 60.7% models on disk keep their filenames and stay loadable as the fallback. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:05:14 -04:00
//--- True class base rates (natural, un-oversampled), measured from the label distribution each era
//--- (UpdateClassPriors, EMA-blended for stability) and PERSISTED alongside the weights (.stats
//--- sidecar) so live inference - including after a restart, when no training re-runs - calibrates
//--- exactly as training did. 0 = not yet measured => AdjustedSignalFromSoftmax falls back to raw.
double m_priorBuy, m_priorSell, m_priorNeutral;
//--- Per-era OOS "fired" confusion counts under the LIVE decision rule: a directional call is counted
//--- whenever the prior-corrected posterior is non-Neutral, i.e. exactly the bars on which the
//--- deployed EA would cast a directional vote. m_oosBuyFiredHits/m_oosBuyFired = live Buy precision;
//--- likewise Sell. This is the metric that predicts forward-trading performance (recall/argmax-
//--- precision above score the RAW argmax, before the base-rate correction the live decision applies).
//--- Reset each era. Whether a counted vote also clears Min_Vote_Open against the other filters'
//--- average is an aggregate question this per-bar scorer cannot see - see its use site.
int m_oosBuyFired, m_oosBuyFiredHits;
2026-07-30 11:47:15 -04:00
//--- The same live-fired population as above, but BUCKETED BY CONFIDENCE TIER (ConfidenceTier(), 4
//--- buckets quartiled from the head's structural floor). Exists to answer the one question the
//--- aggregate precision cannot: whether raising Min_Vote_Open would actually buy precision, and how
//--- much coverage it would cost. Tier weights are 25/50/75/100, and for an AI-only configuration the
//--- averaged vote IS the tier weight, so these four rows map directly onto the input: a floor of 50
//--- keeps tiers 1-3, 75 keeps 2-3, 100 keeps tier 3 alone. Measuring it beats guessing at it - the
//--- floor is only worth raising if precision actually rises monotonically across the tiers, and if it
//--- does not, that is itself the finding (the model's confidence is not calibrated to correctness).
int m_oosTierFired[4], m_oosTierHits[4];
int m_oosSellFired, m_oosSellFiredHits;
//--- Cumulative (compounded, persistent) DIRECTIONAL accuracy = the win-rate of the model's Buy/Sell
//--- calls: of the bars it actually called Buy or Sell, how many matched the true label. Neutral ("no
//--- trade") calls are deliberately EXCLUDED - counting them inflates the rate to ~80%+ (Neutral is the
//--- ~94% majority the model gets right for free) and tells you nothing about trade quality. Summed
//--- across every era AND carried across restarts via the .stats sidecar (WST5); never reset era-to-era,
//--- so the panel shows a stable win-rate that only firms up as more pivots confirm. Incremented at the
//--- IS (pass 2) and OOS (pass 3) hit sites, only when the PREDICTION is directional, skipped for
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- Reset only by ResetWeights (a fresh model).
long m_cumIsCorrect, m_cumIsTotal;
long m_cumOosCorrect, m_cumOosTotal;
//--- Latest live-fired precision (%) and fire count per direction (-1 = n/a), cached at era end for
//--- the status panel/log the same way m_lastBuyRecallPct is (see its comment).
int m_lastBuyFiredPrecPct, m_lastSellFiredPrecPct;
int m_lastBuyFired, m_lastSellFired;
//--- Ceiling on the class-balance sampleWeight multiplier (see its computation in Train(), keyed off
//--- m_prevEraTrueBuyCount/Sell/Neutral). MAX_WEIGHT_DELTA (AI\Network.mqh/.cl, DirectML\WarriorCPU.cpp)
//--- bounds how far a SINGLE weight can move in one step, but it does nothing to stop many small
//--- steps in the same direction from accumulating into a large net swing across an era - with real
//--- label counts around Buy 1340 / Sell 1312 / Neutral 7013 the uncapped ratio is ~5.3x on every
//--- single Buy/Sell example, every era. At the old hardcoded 3.0x ceiling that was STILL strong
//--- enough in practice to let one era's Buy gradients overwrite the separability the previous era
//--- had just built for Sell (and vice versa) - the observed anti-correlated Buy/Sell recall whipsaw
//--- (e.g. era 8 Buy:0%/Sell:26% -> era 9 Buy:10%/Sell:57% -> era 10 Buy:1%/Sell:37%), never both
//--- classes improving together. Was tunable (ClassSampleWeight input, CSW_15/1.5x default) as a
//--- loss-level multiplier, but that mechanism proved structurally too weak against Adam's near-
//--- invariance to constant gradient rescaling (Kingma & Ba 2015) - see the m_isTrainQueue queueing
//--- block's oversampling comment for the full history. Class-balance correction now happens
//--- entirely via data-level oversampling (repCount in that queueing block); this member is
//--- currently unread by that path. The ClassSampleWeight INPUT that used to set it was removed
//--- (it had no effect); the member is kept at its constructor default in case a smaller, additive
//--- loss-level nudge is ever reintroduced on top of oversampling.
double m_maxClassSampleWeight;
//--- FOCAL LOSS REMOVED 2026-07-31 (was m_focalGamma / the FocalLossGamma input). Lin et al. 2017's
//--- (1-pt)^gamma is sound, but it corrects the SAME axis as the logit-adjusted loss, and stacking
//--- two corrections on one axis is the failure Buda et al. 2018 warns about (cited in the queueing
//--- block). It was also running at gamma*0.125, damped by a replay flag whose path was already dead.
//--- The ladder's escape is the learning-rate warm restart; the gamma anneal was a monotone step to
//--- zero, so nothing went with it. Full nine-input audit: the class-imbalance block in Inputs.mqh.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- ZigZag repainting embargo, in bars. The real MQL5 ZigZag (CustomIndicators\ADZigZag.mq5 - a
//--- renamed, logic-untouched copy of the stock ZigZag.mq5 at its stock defaults: Depth=12,
//--- Deviation=5 points, Backstep=3) revises its most recent 1-3 legs as new bars arrive (see
//--- ADZigZag.mq5's own ExtRecalc), so a bar's ADZigZagBuffer value is only trusted once at least
//--- this many MORE bars have closed after it.
//--- SCOPE NARROWED 2026-08-01: this used to gate the training LABELS as well, back when the target
//--- was the exact confirmed pivot. The target is now the triple barrier, whose lookahead is its own
//--- horizon (m_barrierHorizonBars), so this constant survives for exactly one job - keeping the
//--- swing-context INPUT FEATURES (m_useSwingContext) from reading a leg that can still change.
int m_swingConfirmationBars;
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- Vertical barrier of the triple-barrier label, in bars - see BARRIER_HORIZON_LADDER_COUNT. Derived
//--- once by ComputeBarrierHorizonBars() at the start of the label prebuild and then held for the run.
//--- It is ALSO the label's lookahead depth, so it is what the IS/OOS embargo and the online-learning
//--- confirmation delay must both wait out: a bar's barrier label is not knowable until this many more
//--- bars have closed after it. That job used to belong to m_swingConfirmationBars, which answered the
//--- ZigZag question ("has this leg stopped repainting") and no longer answers the label's.
int m_barrierHorizonBars;
fix: refuse invalid SL/TP, fix the unreachable deploy floor, scale the horizon Three defects found by reading the 2026-08-01 training logs, all of which only became visible because the relabel made the numbers mean something. 1. A STALE ENUM TRAINED FOUR MODELS ON THE WRONG TARGET. `OnInit: trade settings snapshot - SL_Mode=1 TP_Mode=-101` -101 was TP_PREV_SWING, deleted from TAKE_PROFIT_MODE on 2026-07-31 in 7eb48f5. MetaTrader does not validate a saved enum input against the enum's current members, so charts saved before that kept the old integer. BarrierMultiples()'s `if(tpMult <= 0.0) tpMult = slMult;` then quietly turned it into a 1:1 barrier, and all four topologies trained ~250 eras against a strategy nobody selected - while the log reported "target 1.00*ATR" as though it were configured. Since the relabel these two inputs ARE the label definition, so this is not a bad trade setting, it is a wrong dataset. ValidateBarrier- Inputs() now refuses to start (INIT_FAILED + Alert + an explicit fix) on any value that is not an enum member. Members are enumerated rather than range-checked because both enums are sparse and carry negative sentinels, so no min/max test can tell a legal value from a deleted one - which is the entire failure mode. The fallback survives as belt-and-braces but now announces itself: a fallback that cannot say it fired is indistinguishable from correct behaviour. 2. THE DEPLOYABILITY FLOOR BECAME MATHEMATICALLY UNREACHABLE. `tradeableOK` required `dirPrecPct >= baseRatePct`, where baseRatePct is Buy+Sell as a share of all bars. At the old exact-pivot target that was ~6%, so "beat the base rate" read as "beat chance" and the test looked sound. Triple-barrier labels put it at ~83%, so the gate now demanded 83% directional precision - impossible by construction. Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing safe to deploy" at a perfectly healthy 43-45% precision, with no checkpoint able to ship however good it got. Replaced with ZERO-SKILL precision, max(Buy,Sell)/allBars: exactly the score of the degenerate always-call-one-direction model this floor exists to reject. Correct at any base rate - ~43% on the current labels, ~3% on the old rare-pivot ones. The era line now prints "(chance N%, edge +Mpp)" beside the selection score, because 44% precision is excellent against a 3% chance level and worthless against a 43% one, and reading the first as the second is what made tonight's run look better than it was. 3. THE HORIZON IGNORED THE BARRIER GEOMETRY. ComputeBarrierHorizonBars() returned the median ZigZag leg, which measures how long a ~1 ATR move takes and says nothing about how long the CONFIGURED barrier needs. First-passage time out of [-m,+k] scales with m*k, so a 1:3 barrier takes ~3x as long as 1:1; the unscaled horizon would have timed out most 1:3 trades and pushed Neutral straight back up, re-creating the imbalance the relabel removes. Now multiplied by slMult*tpMult, calibrated against a real measurement rather than assumed: the accidental 1:1 run resolved at horizon 12 with only 16.7% timeouts, so the swing median is the right scale at m*k=1. Verifiable, not just asserted: the prebuild now counts barriers that ended on the VERTICAL barrier and reports them as a share of Neutral. Neutral conflates "timed out" with "stopped out" and only the first indicts the horizon. Both builds compile 0 errors / 0 warnings. Forces a retrain - correcting TP_Mode re-keys the fingerprint (|TB:1:-101 -> |TB:1:3), which is right: no existing model was trained on the intended target. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:30:49 -04:00
//--- Latch so the invalid-TP fallback in BarrierMultiples() shouts once, not once per labelled bar.
bool m_barrierFallbackWarned;
//--- Did the LAST TripleBarrierLabel() call run out of horizon without either barrier being touched?
//--- Neutral conflates two very different outcomes - "the trade timed out" and "the stop was hit
//--- before the target" - and only the first one indicts the horizon. Counting them apart is what
//--- lets the m*k horizon scaling in ComputeBarrierHorizonBars() be VERIFIED against a real run
//--- rather than trusted: a high timeout share means the horizon is too short for the configured
//--- barrier, a high stop-out share just means the barrier is hard.
bool m_lastBarrierTimedOut;
fix: both-won bars were labelled "do not trade" - resolve by first touch Removing the min-reward:risk raise let the MEASURED geometry come back with the target NEARER than the stop (SP500 H1: target 1.62*ATR at q50 of favourable, stop 3.33*ATR at q75 of adverse). That reopened a branch the code called unreachable: price can reach +target and -target inside one horizon, winning in BOTH directions, and those bars fell through to Neutral. Neutral has only three producers, both-lost is unreachable (you cannot touch -3.33 without crossing -1.62 first, which wins the short), and timeouts logged at 1.0% of Neutral - so ~27% of ALL bars were being handed to the model as the abstain class when a trade either way would have collected its target. The cleanest positives in the sample, labelled "do not trade", while the fitted confidence threshold was being asked to find selectivity in what was left. Resolved by FIRST TOUCH: the target reached earlier is the trade that would have closed first. Same forward window, no extra lookahead. Same-bar ties stay Neutral - OHLC cannot order two touches, and unlike an intrabar stop tie there is no pessimistic side to fall to, so a guess would inject a coin-flip direction into the target. Also: - count both-won and its same-bar tie subset in the prebuild line, so the share is measured rather than inferred from arithmetic on a log line - scope the timeout counter to IS, matching the tally it is reported as a percentage OF; it was incremented over the whole scan and divided by an in-sample denominator - clear m_lastBarrierTimedOut at the top of the walk with the excursions, not at the bottom - the two early returns published the previous bar's verdict - mark the pass-1 label line PROVISIONAL. It prints the enum fallback because geometry can only be derived from excursions that do not exist yet, and it reads exactly like a config change that failed to take effect FORCES RETRAIN. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:35:42 -04:00
//--- Did the LAST call find BOTH directions' targets reachable inside the horizon? Impossible while
//--- reward >= risk - the case only opened up when the minimum-reward:risk raise was removed and the
//--- MEASURED geometry came back with the target (q50 of favourable travel) CLOSER than the stop (q75
//--- of adverse). It is not an edge case there: it is the whipsaw class, and on SP500 H1 it accounts
//--- for nearly all of Neutral. Published so the prebuild can count it - see
//--- m_labelPrebuildBothWonCount for why a bar that wins in either direction must not be labelled
//--- "do not trade".
bool m_lastBarrierBothWon;
//--- Subset of the above where both targets fell inside the SAME bar, so OHLC cannot say which came
//--- first. Those stay Neutral, for the same reason intrabar ties score as the stop: the file refuses
//--- to order two touches it cannot see the order of.
bool m_lastBarrierBothWonTied;
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
//--- Did a LONG / a SHORT placed at this bar reach its target before its stop? These are the raw
//--- questions the barrier walk answers, before they are collapsed into one 3-class label, and they
//--- are what a trade's profitability actually depends on. Kept separately because the collapse is
//--- LOSSY in exactly the case that now matters: on a both-won bar the label names one direction, but
//--- BOTH trades would have paid, so scoring the other call as an error understates the model. See
//--- m_oosWinLongTotal for why the deploy gate had to stop using label agreement as its hit test.
bool m_lastWinLong;
bool m_lastWinShort;
diag: is "optimal SL/TP" learnable? Score the features against excursions Proposed direction: train the net to predict entry/SL/TP that maximise return and minimise drawdown, rather than to classify direction. Before rebuilding a head, measure whether the target is learnable at all. That question splits into two that behave nothing alike: HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility clustering is about the most robust regularity in markets. WHICH WAY it goes first (the asymmetry) - direction, which is what every noise-floor verdict in this project has been about. Expectancy comes ONLY from the second. The first buys position sizing and drawdown control - worth having under prop-firm limits, but not an edge: exit management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with expectancy FLAT. Crucially this is NOT already answered. Every MI figure here scored the triple-barrier label, i.e. one specific question at one fixed geometry. A noise-floor result there says nothing about whether excursion MAGNITUDE is learnable - different target, different answer. Four targets, and the verdict is the CONTRAST, printed explicitly because the dangerous misreading of "UP clears" is "we can predict profitable trades": RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that SHOULD clear. Every prior verdict here lacked a control expected to pass; a range target at the floor indicts the measurement, not the market. UP / DOWN - MFE / MAE. ASYMMETRY - up-dn, the only one that can pay. Collected inside the walk the label already does (one max, one min per bar). The early-out when both barriers resolved is GONE: it would have truncated the excursions at whichever bar tripped the last barrier, making the measurement a function of the CURRENT SL/TP - the circularity this is trying to escape. The loop was already bounded by the horizon, so only the average cost moves. Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is fat-tailed and fixed-width bins would put nearly every row in bin 0; it also pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and to the barrier label's ~1.02 instead of confounded by class balance. Two bugs fixed in this code before it ever ran, both of which would have produced a plausible quiet wrong answer rather than an error: - TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the accumulators were reset, so one bar's excursions would be cached under another bar's index. Cleared at the top now, ahead of every return. - An unresolvable bar is still flagged as labelled but carries excursions of exactly 0. Under equal-frequency binning a block of identical zeros drags the lowest cut onto zero and a third of the sample lands in one uninformative bin - a depressed score that reads as "not predictable", a false negative in the direction that would wrongly kill the idea. Rows where both excursions are zero are dropped; price cannot travel zero both ways over a whole horizon. Read-only diagnostic. No topology or label change: no retrain of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 10:22:41 -04:00
//--- Excursions of the bar TripleBarrierLabel() just resolved, in ATR units, published the same way
//--- m_lastBarrierTimedOut is: the walk that finds them is the walk the label already does, so they
//--- cost one max and one min per bar rather than a second pass over history.
double m_lastExcUp; // (maxHigh - entry)/ATR over the horizon, >= 0
double m_lastExcDown; // (entry - minLow)/ATR, >= 0
feat: derive the ATR multiples from measured excursions - no hardcoded geometry The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in 3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of eleven guesses is not deriving anything. WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It ranks pairings by how predictable their OUTCOME is - a question about direction. The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x. Hence the scan failing its own gate on every run, and its "winner" wandering 2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is strongly measurable, so derive the geometry from that instead. stop = q25 of measured ADVERSE travel (ordinary noise does not reach it) target = q50 of measured FAVOURABLE travel (reached ~half the time, by construction, inside the horizon) Continuous, in ATR units, superseding the enum multiples. Reachability ("target on X% of bars, stop on Y%") and the implied break-even are printed so the choice is auditable rather than trusted. FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the horizon with the target (first-passage time grows with the band) and the excursions are measured OVER the horizon, so target -> horizon -> excursions -> target is a real loop - deriving once sizes the target from travel measured under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at 3 passes, and says so if it does not settle. Does NOT create expectancy, and the log says as much: chance precision equals break-even at every geometry (m/(m+k) on both sides). It buys a target the market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio forces a target the market rarely reaches, it WARNS rather than overriding - the ratio is the user's risk policy, so the honest move is to state its cost. That is the collision that once rejected 100% of setups. Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg files written earlier today still load (their length guard finds no doubles) and a model that carries them was trained on them and never re-derives. Also fixes a message from e5ceed6 that claimed "this model resumed from disk" unconditionally - it printed above a "seeding era 0" line on a brand-new model, because the branch fires whenever the cache is not built, which is equally true before a fresh model's first prebuild. A diagnostic that misreports its own trigger is worse than one that says nothing: it gets quoted back as evidence. FORCES A FULL RETRAIN (labels change). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 12:06:25 -04:00
//--- DERIVED barrier multiples, in ATR units, taken from the measured excursion distribution rather
//--- than from an enum. Zero means "not derived yet" and BarrierMultiples() falls back to the mode
//--- constants. Continuous on purpose: the whole point is to stop snapping the geometry to {2,3} x
//--- {2,3,4,6,8,10}, a grid whose members were guesses.
double m_derivedSlMult;
double m_derivedTpMult;
bool m_geometryDerived;
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//--- DIRECTIONAL CONFIDENCE THRESHOLD - see DIR_CONF_THRESHOLD_BINS for the rationale. The ACTIVE
//--- value, read by AdjustedSignalFromSoftmax() on every live and pass-3 decision; 0.0 means
//--- unthresholded (era 0, or an IS histogram too sparse to fit). Refitted at the end of every
//--- pass 2 from that era's own IS margins, because the margin distribution moves with the weights.
double m_dirConfThreshold;
//--- The value that belongs to the CHECKPOINTED weights. Captured at the same instant as
//--- Net.CaptureWeights() and restored beside them, because a threshold fitted for one set of
//--- weights is meaningless against another - deploying era 40's weights under era 63's operating
//--- point would silently change both coverage and precision away from the numbers the deploy gate
//--- actually cleared.
double m_bestDirConfThreshold;
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- Margin histogram for the fit, rebuilt each era from the CALIBRATION slice (see
//--- DIR_CONF_CALIB_PCT_OF_IS). Index = bin of (winning direction's probability - its best rival's);
//--- [0] counts directional calls, [1] counts the ones whose implied trade WON. No oversampling
//--- applies here - the calibration walk visits each bar once, chronologically - so the primary-only
//--- correction the replay queue needed (see m_cumIsTotal) is structural rather than a filter.
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
long m_dirConfBinCalls[DIR_CONF_THRESHOLD_BINS];
long m_dirConfBinHits[DIR_CONF_THRESHOLD_BINS];
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
long m_dirConfPrimaryBars; // denominator for coverage: every calibration bar scored
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//--- one-shot so the "histogram too sparse" explanation is stated once per run, not once per era
bool m_dirConfSparseWarned;
feat: derive the ATR multiples from measured excursions - no hardcoded geometry The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in 3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of eleven guesses is not deriving anything. WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It ranks pairings by how predictable their OUTCOME is - a question about direction. The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x. Hence the scan failing its own gate on every run, and its "winner" wandering 2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is strongly measurable, so derive the geometry from that instead. stop = q25 of measured ADVERSE travel (ordinary noise does not reach it) target = q50 of measured FAVOURABLE travel (reached ~half the time, by construction, inside the horizon) Continuous, in ATR units, superseding the enum multiples. Reachability ("target on X% of bars, stop on Y%") and the implied break-even are printed so the choice is auditable rather than trusted. FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the horizon with the target (first-passage time grows with the band) and the excursions are measured OVER the horizon, so target -> horizon -> excursions -> target is a real loop - deriving once sizes the target from travel measured under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at 3 passes, and says so if it does not settle. Does NOT create expectancy, and the log says as much: chance precision equals break-even at every geometry (m/(m+k) on both sides). It buys a target the market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio forces a target the market rarely reaches, it WARNS rather than overriding - the ratio is the user's risk policy, so the honest move is to state its cost. That is the collision that once rejected 100% of setups. Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg files written earlier today still load (their length guard finds no doubles) and a model that carries them was trained on them and never re-derives. Also fixes a message from e5ceed6 that claimed "this model resumed from disk" unconditionally - it printed above a "seeding era 0" line on a brand-new model, because the branch fires whenever the cache is not built, which is equally true before a fresh model's first prebuild. A diagnostic that misreports its own trigger is worse than one that says nothing: it gets quoted back as evidence. FORCES A FULL RETRAIN (labels change). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 12:06:25 -04:00
int m_geometryDerivePasses; // fixed-point iteration counter, capped
fix: a restart no longer loses the measured geometry or the training window Terminal restart, 22:25: all four resumed models sat on empty windows with enum 2:6 barriers. Three interlocking causes, all visible in one log excerpt: 1) THE PRE-SCAN WINDOW WAS SIZED BY THE SAVED WATERMARK. A resumed model's dtStudied sits at its last studied bar, so Bars(dtStudied, now) ~ 0 and the resumed-model MI pre-scan built a zero-bar "complete" label cache - logged as "Buy: 0 | Sell: 0 | Neutral: 0". Train()'s own era start RESETS dtStudied to the training-window rule before computing its window; the pre-scan did not. The rule is now factored into TrainWindowStart() and both use it. The scan also refuses to arm before SERIES_SYNCHRONIZED (it ran in the same second as OnInit), and deployed models keep their watermark - for them it gates inference recency, not a training window. 2) THE HORIZON LATCHED ON AN INDICATOR WARM-UP. ComputeBarrierHorizonBars ran against a ZigZag with 0 calculated legs, fell back, and EnsureBarrierHorizon latched fallback(32) x slMult x tpMult = 384 for the process lifetime. A leg-starved horizon is now PROVISIONAL: re-resolved on the next rebuild, the label cache wiped if it moved (labels from two horizons answer different questions), and the geometry deriver refuses to run from it - a pair derived over a warm-up window would get PINNED. 3) THE DERIVED GEOMETRY WAS NEVER PERSISTED. The .cfg is written at model creation and at weights-reset - both BEFORE era 0 derives - so the measured pair lived only in memory: every restart read back zeros, adopted nothing, fell back to the enum barriers, and the era-0-only gate meant a resumed model could NEVER re-derive. A full day of training on 3.33/1.62 resumed as 2:6. Now: the settled pair is pinned to the .cfg the moment derivation completes (one-shot, atomic write), and the derive gate accepts any model with no pinned pair, not just era 0 - mid-run stability is carried by m_geometryDerived itself, which never allows a second derivation. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 22:40:43 -04:00
//--- The last ComputeBarrierHorizonBars() ran with FEWER confirmed ZigZag legs than the median
//--- needs, so the horizon it returned is the fallback, not a measurement. While this is true the
//--- horizon stays PROVISIONAL (m_barrierHorizonResolved is left false so the next full rebuild
//--- re-resolves it) and the geometry deriver refuses to run - deriving a permanent, .cfg-pinned
//--- barrier from excursions measured over a made-up window would pin the artifact, not the market.
//--- Observed 2026-08-09 22:25: a terminal restart ran the resumed-model pre-scan in the same
//--- second as OnInit, the ZigZag had 0 calculated legs, and the horizon latched to fallback(32) x
//--- slMult x tpMult = 384 for the whole process.
bool m_barrierHorizonLegStarved;
bool m_horizonStarvedWarned; // one-shot: the starved path can retry every call
//--- Has THIS process written the derived geometry into the .cfg? The .cfg was only ever written at
//--- model creation and at weights-reset - both BEFORE era 0's derivation - so the derived pair
//--- never reached disk, every resumed model read back zeros, adopted nothing, fell back to the
//--- enum barriers, and (era > 0) never re-derived: trained on 3.33/1.62 all day, relabelled at 2:6
//--- on the next restart. Set by the post-derivation save, and also by the adoption path (the pair
//--- is already on disk there).
bool m_geometryCfgSaved;
fix: excursion window must not depend on the barrier it sizes DIRECTION IS NOT THERE, and this run is what establishes it. Three symbols: raw ASYMMETRY clears on all three (p=0.0199 / 0.0050 / 0.0050) norm ASYMMETRY collapses on all three (p=0.3433 / 0.5075 / 0.2736), USDCAD landing BELOW its own null RANGE control strengthens to 3-5x its null everywhere Divide sigma out and the apparent directional signal vanishes entirely. What cleared was volatility leaking through an unnormalised difference. Note this would have passed any replication test: three instruments at p=0.005 is exactly the evidence one would accept before committing to a rebuild, and the confound reproduces perfectly. Replication was never going to catch it - only the normalisation could. Two defects of mine, both surfaced by the same run. 1. THE GEOMETRY DERIVATION WAS DIVERGING, NOT CONVERGING. It produced a 14.57*ATR stop and a 29.14*ATR target that only 5.7% of bars ever reach. Excursions were measured over the barrier horizon; the horizon scales with the target; the target is a quantile of the excursions - so target -> horizon -> excursions -> target ran away, and "settled" only because the horizon ladder caps at 384 bars. A saturated runaway, which the iteration guard could not catch because it watches for OSCILLATION. Fixed at the root: excursions now accumulate only over m_swingMedianBars - the UNSCALED median ZigZag leg, a property of the instrument that owes nothing to the barrier. The barrier walk still runs the full horizon, because that is how long the trade is held; only the MEASUREMENT used to size the barrier is confined to a geometry-independent window. (The Min_Risk_Reward_Ratio warning fired correctly and is what flagged it - the diagnostic worked while the derivation behind it did not.) 2. THE CONFOUND VERDICT WAS UNREACHABLE. `sizeCleared && !asymCleared` was tested first and is true whenever size clears - i.e. always - so the branch that NAMES the volatility confound never printed; all three symbols showed the generic size-not-direction message instead. Verdict chain rewritten with the specific case first, and the dangling elses my first patch introduced removed. FORCES A FULL RETRAIN (the excursion window changes every derived barrier). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 13:57:23 -04:00
//--- Median confirmed ZigZag leg in bars, UNSCALED by the barrier. The window excursions are measured
//--- over, kept independent of the geometry so sizing the geometry from them cannot feed back.
int m_swingMedianBars;
fix: refuse invalid SL/TP, fix the unreachable deploy floor, scale the horizon Three defects found by reading the 2026-08-01 training logs, all of which only became visible because the relabel made the numbers mean something. 1. A STALE ENUM TRAINED FOUR MODELS ON THE WRONG TARGET. `OnInit: trade settings snapshot - SL_Mode=1 TP_Mode=-101` -101 was TP_PREV_SWING, deleted from TAKE_PROFIT_MODE on 2026-07-31 in 7eb48f5. MetaTrader does not validate a saved enum input against the enum's current members, so charts saved before that kept the old integer. BarrierMultiples()'s `if(tpMult <= 0.0) tpMult = slMult;` then quietly turned it into a 1:1 barrier, and all four topologies trained ~250 eras against a strategy nobody selected - while the log reported "target 1.00*ATR" as though it were configured. Since the relabel these two inputs ARE the label definition, so this is not a bad trade setting, it is a wrong dataset. ValidateBarrier- Inputs() now refuses to start (INIT_FAILED + Alert + an explicit fix) on any value that is not an enum member. Members are enumerated rather than range-checked because both enums are sparse and carry negative sentinels, so no min/max test can tell a legal value from a deleted one - which is the entire failure mode. The fallback survives as belt-and-braces but now announces itself: a fallback that cannot say it fired is indistinguishable from correct behaviour. 2. THE DEPLOYABILITY FLOOR BECAME MATHEMATICALLY UNREACHABLE. `tradeableOK` required `dirPrecPct >= baseRatePct`, where baseRatePct is Buy+Sell as a share of all bars. At the old exact-pivot target that was ~6%, so "beat the base rate" read as "beat chance" and the test looked sound. Triple-barrier labels put it at ~83%, so the gate now demanded 83% directional precision - impossible by construction. Observed live: all four topologies cycling "PLATEAU stage 3 ... nothing safe to deploy" at a perfectly healthy 43-45% precision, with no checkpoint able to ship however good it got. Replaced with ZERO-SKILL precision, max(Buy,Sell)/allBars: exactly the score of the degenerate always-call-one-direction model this floor exists to reject. Correct at any base rate - ~43% on the current labels, ~3% on the old rare-pivot ones. The era line now prints "(chance N%, edge +Mpp)" beside the selection score, because 44% precision is excellent against a 3% chance level and worthless against a 43% one, and reading the first as the second is what made tonight's run look better than it was. 3. THE HORIZON IGNORED THE BARRIER GEOMETRY. ComputeBarrierHorizonBars() returned the median ZigZag leg, which measures how long a ~1 ATR move takes and says nothing about how long the CONFIGURED barrier needs. First-passage time out of [-m,+k] scales with m*k, so a 1:3 barrier takes ~3x as long as 1:1; the unscaled horizon would have timed out most 1:3 trades and pushed Neutral straight back up, re-creating the imbalance the relabel removes. Now multiplied by slMult*tpMult, calibrated against a real measurement rather than assumed: the accidental 1:1 run resolved at horizon 12 with only 16.7% timeouts, so the swing median is the right scale at m*k=1. Verifiable, not just asserted: the prebuild now counts barriers that ended on the VERTICAL barrier and reports them as a share of Neutral. Neutral conflates "timed out" with "stopped out" and only the first indicts the horizon. Both builds compile 0 errors / 0 warnings. Forces a retrain - correcting TP_Mode re-keys the fingerprint (|TB:1:-101 -> |TB:1:3), which is right: no existing model was trained on the intended target. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 00:30:49 -04:00
int m_labelPrebuildTimeoutCount;
fix: both-won bars were labelled "do not trade" - resolve by first touch Removing the min-reward:risk raise let the MEASURED geometry come back with the target NEARER than the stop (SP500 H1: target 1.62*ATR at q50 of favourable, stop 3.33*ATR at q75 of adverse). That reopened a branch the code called unreachable: price can reach +target and -target inside one horizon, winning in BOTH directions, and those bars fell through to Neutral. Neutral has only three producers, both-lost is unreachable (you cannot touch -3.33 without crossing -1.62 first, which wins the short), and timeouts logged at 1.0% of Neutral - so ~27% of ALL bars were being handed to the model as the abstain class when a trade either way would have collected its target. The cleanest positives in the sample, labelled "do not trade", while the fitted confidence threshold was being asked to find selectivity in what was left. Resolved by FIRST TOUCH: the target reached earlier is the trade that would have closed first. Same forward window, no extra lookahead. Same-bar ties stay Neutral - OHLC cannot order two touches, and unlike an intrabar stop tie there is no pessimistic side to fall to, so a guess would inject a coin-flip direction into the target. Also: - count both-won and its same-bar tie subset in the prebuild line, so the share is measured rather than inferred from arithmetic on a log line - scope the timeout counter to IS, matching the tally it is reported as a percentage OF; it was incremented over the whole scan and divided by an in-sample denominator - clear m_lastBarrierTimedOut at the top of the walk with the excursions, not at the bottom - the two early returns published the previous bar's verdict - mark the pass-1 label line PROVISIONAL. It prints the enum fallback because geometry can only be derived from excursions that do not exist yet, and it reads exactly like a config change that failed to take effect FORCES RETRAIN. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:35:42 -04:00
//--- Bars where BOTH targets were reached, and the same-bar subset that could not be ordered. Counted
//--- rather than inferred: the share was first arrived at by subtracting timeouts from Neutral and
//--- reasoning that both-lost is unreachable, which is sound but is still arithmetic on a log line.
//--- These two numbers make the label composition readable directly, and the tie count is the one that
//--- says whether first-touch resolution is doing real work or just relabelling noise.
int m_labelPrebuildBothWonCount;
int m_labelPrebuildBothWonTieCount;
//--- safety valve: Train()'s do-while loop has no other bound on how many eras it will run
//--- before giving up, so a config that can't reach the convergence objective (e.g. too few
//--- swing-confirmed examples for the min recall bar to be reachable) would otherwise loop
//--- forever, permanently keeping the era-progress status label up instead of the normal per-tick
//--- info line and burning CPU nonstop. When the cap is hit, the operator is prompted (see
//--- PromptContinuePastEraCap): CONTINUE resets the era counter and keeps training; STOP deploys
//--- the best checkpoint found so far (FinalizeTrainRun) and terminates training. Headless
//--- (tester/optimizer) runs can't prompt, so they take the STOP branch automatically.
int m_maxErasPerRun;
//--- Train() runs its per-bar loop synchronously, and MQL5 is single-threaded per chart - a
//--- multi-minute era would otherwise starve the terminal's chart-event queue for that whole
//--- stretch, including the control panel's own click/drag hit-testing (Panel\ControlPanel.mqh),
//--- which depends entirely on CHARTEVENT_MOUSE_MOVE being delivered promptly. Train() is
//--- therefore chunked: each call does at most ~TRAIN_TIME_BUDGET_MS of work, then returns and
//--- picks back up exactly where it left off on the next call (re-triggered via the same "New
//--- Bar" custom-event scheduling ScheduleTrainingIfNeeded() already used) - the members below
//--- are what makes that resumable across calls.
bool m_trainRunActive; // true: a run (schedule -> convergence/stop) is in progress, possibly spanning many Train() calls
bool m_eraResumePending; // true: yielded mid-bar-loop last call - resume the SAME era, don't start a new one
int m_resumeBars;
int m_resumeTotalIter;
int m_resumeOosCutoff;
int m_resumeBarIndex;
bool m_resumeAddLoop;
//--- Pass 2 of the era loop: bar indices pass 1 (sequential feedForward/scoring) queued as
//--- IS-eligible for backProp, trained on in a freshly shuffled order instead of pass 1's own
//--- fixed chronological (oldest-to-newest) visitation order. Root cause this targets: Adam's
//--- momentum (b1, AI\Network.mqh) has an effective memory window of ~1/(1-b1) steps, which lands
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- close to the length of the contiguous same-class label runs the target produces (triple-barrier
//--- labels make those runs LONGER than the exact-pivot ones this was first measured against, since
//--- adjacent bars share most of their forward window and usually resolve the same way, so the
//--- argument holds a fortiori) - replaying those runs in the SAME order every single era let momentum lock onto
//--- whichever run it was currently passing through, with the network's end-of-era state
//--- disproportionately reflecting whatever it last walked through (recency), not a globally
//--- balanced fit. Symptom observed in practice: IS error climbing era-over-era (0.16->0.55+ over
//--- 25 eras on the same fixed dataset) instead of settling, and OOS Buy/Sell recall whipsawing
//--- between near-0% and 70-100% despite near-equal Buy/Sell label counts. Shuffling is the
//--- standard SGD fix - see the m_isPass2Active declaration below for how it's sequenced against
//--- pass 1 within Train()'s existing resumable-chunk machinery.
int m_isTrainQueue[];
int m_isTrainQueueCount;
//--- Parallel to m_isTrainQueue (same index, kept in lockstep through the Fisher-Yates shuffle
//--- below) - the per-occurrence weight to apply when this queued slot is trained on in pass 2,
//--- decided once at queue time (see the queueing block's oversampling comment) rather than
//--- recomputed in pass 2. Currently always 1.0: class-balance correction is carried entirely by
//--- repCount (how many times a bar was duplicated into the queue), not by any per-occurrence
//--- weight scaling - see the queueing block's comment for why stacking a second correction here
//--- caused two separate training collapses. Kept as a real per-slot array rather than a literal
//--- 1.0 so a future, smaller/safer supplemental weight could be reintroduced without re-touching
//--- the queueing or shuffle code.
double m_isTrainQueueWeightScale[];
//--- Parallel to m_isTrainQueue too (same index, same lockstep swap in the shuffle): true on exactly
//--- ONE occurrence per duplicated bar - the rep-0 slot written at queue time. Its only job is to keep
//--- the reported IS accuracy (m_cumIsCorrect/m_cumIsTotal) measured on the NATURAL class distribution
//--- while backProp still trains on the oversampled one.
//--- Why: the queue duplicates each Buy/Sell bar up to repCount times (~21x at the observed 30.7:1
//--- imbalance), so counting every occurrence measured IS accuracy over a set that is ~58% directional,
//--- while the OOS counter measures the real ~6% directional distribution. Both excluded Neutral and
//--- both used the identical formula, so they LOOKED directly comparable - and reported things like
//--- "IS 77% / OOS 12%", which reads as catastrophic overfitting when the two numbers were simply
//--- scored against base rates an order of magnitude apart (77% vs a 58% baseline is a 1.33x lift;
//--- 12% vs a 6.1% baseline is 1.97x - the OOS side was actually the STRONGER one).
//--- Counting primaries only puts both metrics on the same footing, so the IS/OOS gap once again means
//--- what everyone reads it as meaning: generalisation. Training itself is completely unaffected -
//--- every occurrence still backprops exactly as before; this flag is read only by the counter.
bool m_isTrainQueuePrimary[];
//--- Pass 2's cursor into the (already shuffled) m_isTrainQueue - lets pass 2 itself yield/resume
//--- mid-queue under the same TRAIN_TIME_BUDGET_MS chunk budget pass 1 already yields under.
int m_isTrainCursor;
//--- true: pass 1 (sequential) has finished for this era and pass 2 (shuffled backProp) is either
//--- running or has yielded mid-queue - Train() skips straight past pass 1's loop on resume when
//--- this is set. Reset to false only at a fresh era's start (never mid-run).
bool m_isPass2Active;
//--- true: pass 2 has already run to natural completion for this era (m_isPass2Active's own
//--- false state is ambiguous between "not started yet" and "already finished" - both look
//--- identical to a plain `if(!m_isPass2Active)` check). Needed because pass 3 (OOS scoring) can
//--- itself yield/resume across multiple Train() calls same as passes 1/2 do; without this flag,
//--- every resume into an unfinished pass 3 fell through the `if(!m_isPass2Active)` guards on both
//--- pass 1 and pass 2 and re-ran the ENTIRE shuffled queue again (pass 1 itself was a no-op on
//--- resume since its own loop cursor `i` was already exhausted, but pass 2 re-shuffled and replayed
//--- from scratch every single time) - silently multiplying the predicted-class counts (and the
//--- extra, unintended backProp() calls) once per resume, for as long as pass 3 kept needing more
//--- than one chunk to finish. Reset to false only at a fresh era's start, alongside m_isPass2Active.
bool m_isPass2Done;
//--- Pass 3: chronological, OOS-region-only re-walk that happens AFTER pass 2 has actually trained
//--- on this era's IS data - see m_isTrainQueue's declaration comment for why OOS scoring can no
//--- longer just happen inline during pass 1 (that would score every era's OOS window against
//--- weights from BEFORE this era's training, one full era stale - and for era 0 specifically,
//--- against the still-untrained cold-start network, which is why era 0's OOS recall used to show
//--- a meaningless 100% Neutral / 0% Buy / 0% Sell every time). Cursor walks i downward from
//--- m_oosScoreStartIndex to 0, mirroring pass 1's own iteration bounds/order for whichever bars
//--- satisfy isOOS - order matters here (unlike pass 2) since dOosForecast/dOosError are recursive
//--- EMAs over the visitation sequence, not order-independent.
bool m_isPass3Active;
int m_oosScoreIndex;
int m_oosScoreStartIndex;
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- Pass 2.5: the CALIBRATION walk. Sits between pass 2 (train) and pass 3 (grade) because that is
//--- the only position where the operating point can be fitted on data that is neither trained on nor
//--- graded - see DIR_CONF_CALIB_PCT_OF_IS for the measurement that forced it out of pass 2. Chunked
//--- and resumable on exactly the same pattern as passes 1-3 (cursor + active/done flags, both reset
//--- only at a fresh era's start), because at ~5.7k bars it will not finish inside one 120 ms budget.
//--- The band's bounds are recomputed from oosCutoff on every call rather than stashed in resume
//--- state: they are a pure function of (totalIter, oosCutoff, horizon), all three of which the resume
//--- path already restores, so deriving them cannot drift out of step with pass 1's queueing decision.
bool m_isCalibActive;
bool m_isCalibDone;
int m_calibIndex;
int m_calibStartIndex;
feat: excursion-size head (Stage 1, measurement only) Direction is closed - normalised asymmetry fails on three instruments with a working positive control, and the classifier's own best-of-999 era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is a different question and RANGE clears at ~4x its null. Checked the denomination before building on that, since the source memo warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is predictable" is a claim about travel RELATIVE to current ATR, not a restatement of "ATR is autocorrelated". It is exactly the part a fixed multiple (stop 3.31*ATR, target 1.64*ATR) discards. A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches ladder rung k) upward and downward. Survival parameterisation rather than regressing the multiple, because it needs nothing new from CNet: sigmoid outputs and the per-neuron delta the `total != 3` branch already applies (a quantile head would need a linear activation and a pinball gradient in Network.mqh, Network.cl and the DirectML path, on a class four topologies share). Targets are free - m_ladderUpAt already records first-touch age per rung with 0 meaning never reached. Separate net, not extra outputs on the classifier: more outputs would change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push the count off 3 - the exact condition backProp uses to select the joint softmax gradient the 3-class head depends on. The classifier is bit-for-bit unaffected and this is removable without trace. STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the constant per-rung base rate - the baseline a fixed ATR multiple already assumes - with both predictors fitted IS and evaluated OOS, so neither gets a look at the test set. Positive skill justifies Stage 2 (drive SL/TP and sizing off ExcursionQuantile, which is defined and deliberately uncalled). Zero or negative means ATR already carries everything and Stage 2 must not be built. Trains only on primary occurrences: the replay queue oversamples for CLASS balance, and a direction-balanced sample is a biased SIZE sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
//--- EXCURSION-SIZE HEAD. A separate small CNet rather than extra outputs on the classifier: adding
//--- outputs would change m_outputNeuronsCount (branched on in a dozen places), the .nnw shape and
//--- the weights fingerprint, and would push the output count off 3 - which is the exact condition
//--- backProp uses to select the joint softmax gradient the 3-class head depends on. Kept separate,
//--- the classifier is bit-for-bit unaffected and this whole instrument is removable without trace.
//--- NOT PERSISTED in Stage 1: it is a measurement, it retrains within an era or two, and a second
//--- weights file is surface area that only earns its place once the skill score justifies Stage 2.
CNet *m_excNet;
bool m_excHeadFailed; // one-shot: creation failed, do not retry every bar
//--- Allocated once, reused every bar. getResults takes CArrayDouble*& and allocates when handed a
//--- NULL, so locals would mean an allocation per bar across ~32k bars an era.
CArrayDouble *m_excTgt;
CArrayDouble *m_excOut;
long m_excBaseHits[2 * BARRIER_LADDER_COUNT];
long m_excBaseTotal; // rows the base rates were estimated from
double m_excBrierHead[2 * BARRIER_LADDER_COUNT];
double m_excBrierBase[2 * BARRIER_LADDER_COUNT];
int m_excScored; // held-out bars scored this era
fix: the trailing incumbent read the future across eras; cold AD blocks cached zeros as truth Three findings from the 2026-08-11 audit: 1. The excursion head's trailing-quantile ring was deliberately never cleared between eras ("a rolling estimate of the market, not of the era") - but pass 3 re-walks the SAME OOS window every era, so at each walk's restart the ring still held the outcome masks of the newest OOS bars from the previous walk: the chronological FUTURE of the bars about to be scored. For the first ~window+horizon pushes of every era the "trailing" incumbent was partly a leading one - conservative for the gate (an informed incumbent is a harder hurdle) but exactly the self-made-artifact class 06d4785 hunts. The ring now clears at era-score reset; the warm-up bars simply don't score the trail race, which the m_excTrailN gating already accounts for. 2. skillTrail compared the head's FULL-block Brier (pro-rated by coverage) against the incumbent's subset sum - valid only if head skill is uniform across the OOS walk, while the trail-scored subset systematically excludes each era's warm-up bars. The audit also found m_excBrierHeadD/BaseD/ m_excOosHitsD declared, zeroed and never accumulated (dead since e2c9593 made every scored bar disjoint). The dead trio is replaced by m_excBrierHeadT: the head's Brier accumulated only on the bars the warm incumbent also scored, so the race now compares both predictors on an identical bar set. 3. The AD/Wyckoff feature blocks read GetData with no EMPTY_VALUE guard; a cold (still-calculating) indicator returns EMPTY_VALUE everywhere, the sanitize loop rewrote that to 0.0, and the bar SUCCEEDED - so BufferTempData cached an all-zero Wyckoff block as a success for the whole bar frame: the one path the f6150ee only-cache-successes rule cannot see, because it never fails (the ba13eef class, arriving through values that never fail; a resumed model's era-0 prebuild starts milliseconds after OnInit). ADIndicatorCold() probes the NEWEST bar - EMPTY_VALUE there means async warm-up (transient reject, retried), while deep bars beyond the buffered depth keep the sanitize loop's neutral-fill so degraded history still trains. Also fixed m_featureCacheValid's declaration comment, which still described the pre-f6150ee cached-miss semantics. Compile: 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 18:20:24 -04:00
//--- Since e2c9593 every scored bar IS a disjoint window (the score step strides by the horizon),
//--- so m_excBrierHead/m_excOosHits are already the disjoint tally and m_excScoredD just counts it.
//--- The old parallel *D arrays from the two-tally era were declared and reset but never
//--- accumulated - dead weight found by the 2026-08-11 audit and replaced by this one:
//--- the head's Brier accumulated ONLY on the bars the trailing incumbent also scored, so
//--- skillTrail compares the two predictors on the SAME bars instead of pro-rating the head's
//--- full-block sum by coverage (which assumed head skill is uniform across the OOS walk while
//--- the trail-scored subset systematically excludes the warm-up bars).
double m_excBrierHeadT[2 * BARRIER_LADDER_COUNT];
fix: the excursion gate would have passed Stage 2 on an artifact I made Second-opinion review killed the +4.2% far-rung result, correctly, and the mechanism is my own bug. A head trained toward {0.05,0.9} converges to 0.05+0.85p, so its bias is 0.05-0.15p: negative where p is near 1, POSITIVE where p < 1/3, growing monotonically as the rung gets farther. Against a baseline frozen at the IS rate, an upward-biased head scores positive Brier skill whenever the OOS rate merely sits above the IS rate. Predicted signature: huge negatives near, ~zero at p=1/3, growing positives far. Observed: -82% ... -0.6% ... +1.2/+2.7/+4.2. The far rungs were not the clean end of a distorted measurement, they were the other face of the same artifact. Everything before 25aca83 is void. The gate was a bare `skill >= 2%` point estimate over 8 rungs x 4 topologies x N eras, reported per era - a best-of-~300 with no interval and no multiplicity control, which is the shape of the four traps already documented here. It now needs FOUR things at once: DECISION RUNGS only the rungs ExcursionQuantile actually reads at the live geometry (target 1.62, stop 3.31 ATR), fixed before looking. Skill at 5 ATR is skill about a distance no order is placed at - and the TARGET side currently interpolates 1.5/2.0, which measured -2.2% and -1.3%. DISJOINT SAMPLE one bar per horizon. Adjacent bars share 63 of 64 horizon bars, so ~16k scored bars is ~250 independent ones and every SE over the full set is ~8x understated. VS ORACLE the best constant achievable ON THE SCORED BLOCK, closed form from H and n (Brier = H*(1-H/n)). A head that learned only a LEVEL nearer the OOS rate than the frozen IS constant scores positive against the old baseline and <= 0 here. This is the control that separates per-bar skill from base-rate drift. MONOTONE CURVE P(reach k) must be non-increasing in k. Nothing constrained 8 independent sigmoids to obey that, and ExcursionQuantile returns the FIRST crossing - so a tangled curve is misread exactly where the head is least sure. Counted and reported, not silently used. The pass message now also states what a pass would and would not buy: expectancy is -costs at zero directional edge whatever the stop distance, and under prop DD limits LOWER variance also lowers P(reach target before limit), so "better drawdown" is a choice of failure mode, not a win. Still owed before any Stage 2: a race against a trailing-quantile incumbent and a vol-feature logistic. Beating a frozen global constant is the weakest admissible bar for replacing a global constant. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:49:32 -04:00
int m_excScoredD;
//--- OOS positives per rung. Feeds the ORACLE control: the best constant achievable ON THE SCORED
//--- BLOCK, in closed form. Separates "predicts per bar" from "learned a level nearer the OOS rate
//--- than the frozen IS constant", which scores positive while carrying no per-bar information.
long m_excOosHits[2 * BARRIER_LADDER_COUNT];
//--- Bars whose predicted survival curve rose with distance. P(reach k) must be non-increasing in k;
//--- nothing constrains 8 independent sigmoids to obey that, and ExcursionQuantile reads the first
//--- crossing, so a tangled curve is misread exactly where the head is least certain.
int m_excMonoViol;
perf: the excursion head cost 3.6x era time - cut its dispatches ~250x Measured on exc-race-v3: LSTM era 300s -> 1087s (net 272->748s, "other" 30->337s). My estimate had been "single-digit percent". The cost is per-DISPATCH, not per-FLOP, and therefore hits EVERY backend: the head is 19k weights and ~2.4 GFLOP an era - seconds of arithmetic - but ~48k forward/backward calls x several layer submits each, and its 760-wide layer exceeds the CPU DLL's inline threshold so each one pays a real handoff. The classifier's own net time tripled too, from contention with a second pool on an already-full box. Three changes, all backend-neutral because they remove submits rather than tune threads: SCORE ONLY DISJOINT WINDOWS (~64x). Adjacent bars share all but one bar of their horizon, so 16k consecutive bars were always ~250 independent observations - the full-sample tally was never worth more than the disjoint one, it just quoted an n that was ~64x too large. Dropping it costs nothing statistically and removes 63 of every 64 forward passes. The two parallel tallies collapse into one, which is also less code. The trailing ring still advances on every bar: it needs the outcome SEQUENCE, and that is array lookups, not a forward pass. TRAIN ON EVERY 4th PRIMARY BAR (4x). The target is low-dimensional and strongly autocorrelated - neighbouring bars carry near-identical excursion information - so per-bar training buys resolution the target does not have. Strided on ATTEMPTS, not acceptances, so a stretch of unlabelled bars cannot silently change the spacing. OWN TIMING COLUMN. The head's passes were landing in the era line's "other" bucket, which is how a 3.6x regression read as an unexplained jump in the one column nobody attributes. A cost that cannot be seen in the timing line cannot be traded off against anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 16:38:29 -04:00
long m_excTrainTick; // stride counter, on attempts not acceptances
ulong m_excUs; // head's own microseconds this era - see the era line
feat: race the excursion head against a trailing-quantile incumbent Beating a frozen global constant is the weakest admissible bar for replacing a global constant. The honest incumbent is a rolling rung frequency: it adapts to the volatility regime - exactly what the head claims to predict - and needs no model, no 760 inputs and no training. Implemented as a ring of per-bar outcome bitmasks (32 rungs fit one ulong), sized horizon + EXCURSION_TRAIL_WINDOW. The newest `horizon` entries are held back UNRESOLVED: a bar's rung outcomes are only known one horizon later, so using them would be lookahead and would flatter the incumbent into an opponent the head could never fairly beat. Pass 3 walks oldest-to-newest, so "pushed more than horizon bars ago" is exactly "resolved by now". Each push is O(rungs), not O(window). The head's decision-rung Brier is pro-rated to the trailing estimate's coverage before the ratio, since the incumbent only scores bars where its window is warm. This line is worth reading on its own, independently of the head: if the trailing quantile beats the global constant, that is a cheap risk-control win available with no machine learning at all - and it is the same number either way, so the run answers both questions in one pass. The ring is deliberately NOT reset per era - it estimates the market, not the era, and re-warming 500 bars every era would leave the incumbent unusable over the first chunk of every scoring pass, handing the head a free win on exactly those bars. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:57:11 -04:00
//--- TRAILING CLIMATOLOGY (see EXCURSION_TRAIL_WINDOW). Ring of per-bar outcome bitmasks - 32 rungs
//--- fit one ulong, so the whole rolling history is one array of longs. Sized horizon + window: the
//--- newest `horizon` entries are NOT yet resolved (a bar's outcome is only known one horizon later,
//--- and using it would be lookahead), the next `window` entries are the rolling sample the estimate
//--- is taken over, and anything older falls out.
ulong m_excTrailRing[];
int m_excTrailHead; // next write position
int m_excTrailCount; // entries pushed so far, capped at the ring size
long m_excTrailHits[2 * BARRIER_LADDER_COUNT];
long m_excTrailN; // resolved bars currently inside the window
double m_excBrierTrail[2 * BARRIER_LADDER_COUNT];
long m_excTrailScored; // bars scored while the trailing estimate was usable
datetime m_lastBarTime;
//--- This model's own learning-rate trajectory. `eta` (AI\Network.mqh) is a single file-scope
//--- global shared by every CNet in the process - PAI/CONV/LSTM each train independently and
//--- asynchronously (their own Train() calls are interleaved via separate chart-event/timer
//--- scheduling, not synchronized), but all three previously read AND wrote that one global, so
//--- one model's era-end regression/recovery decay/recovery of `eta` (see Train()'s era-end
//--- block) silently changed the learning rate the OTHER two models' very next backProp() call
//--- used too - an unintended coupling between what's supposed to be three independent training
//--- trajectories. Train() now restores `eta` from this member right before touching it, and
//--- saves it back before returning (both at the mid-chunk yield and at the natural end of the
//--- call) - MQL5 gives one chart's EA a single execution thread and each Train() call runs to
//--- completion or to its own yield point before another event is dispatched, so this save/
//--- restore is enough to isolate each model's schedule with no change needed to the neuron-level
//--- or OpenCL/DirectML call signatures that read the global directly.
double m_modelEta;
//--- Ceiling the era-end recovery bump (Train()'s isBetterEra block) restores `eta` toward - used
//--- to be the raw `lr` constant unconditionally, which is only correct for ADAM. SGD's own starting
//--- rate is the separate SgdLearningRate input (AI\Network.mqh) - clamping SGD's recovery bump to
//--- plain `lr` (AdamLearningRate) would silently cap it back down to Adam's ceiling after the
//--- first regression+recovery cycle, undoing the whole point of giving SGD its own configured
//--- rate. Computed once at construction from this model's own m_optimizationAlgo, matching
//--- m_modelEta's per-instance isolation (see that member's comment).
double m_etaCeiling;
int m_erasSinceCooldown; // eras completed since the last cooldown reset - replaces the old per-call-only "erasThisCall"
CArrayDouble m_oosWindow; // run-scoped OOS stability window (used to be a Train()-local CArrayDouble)
double m_bestOosForecast;
//--- Balanced accuracy (macro-recall: mean of Buy/Sell/Neutral OOS recall) of the era the current
//--- checkpoint was taken from. This is the metric the checkpoint SELECTION ranks on now, in place
//--- of the blended dOosForecast - blended accuracy is dominated by Neutral (~96% of bars), so among
//--- otherwise-acceptable eras it silently preferred the MOST Neutral-leaning weights, deploying a
//--- model that scores well on paper but under-calls Buy/Sell. Balanced accuracy weights every class
//--- equally, so "best" now means "most accurate across all three classes" - exactly what a 3-class
//--- signal product wants. Kept SEPARATE from m_bestOosForecast (which still snapshots the blended
//--- value at the same checkpoint) because FinalizeTrainRun()/the restore-on-regression branch reset
//--- dOosForecast to m_bestOosForecast, and that must stay the blended EMA the rest of the machinery
//--- expects. The directional recall FLOOR (directionalRecallOK) still gates convergence unchanged;
//--- this only fixes which era gets deployed among the candidates. -1 until the first ranked era.
double m_bestBalancedOos;
//--- whether the era m_bestOosForecast/the checkpoint was taken from also cleared the per-class
//--- directional recall floor (see directionalRecallOK below) - part of the "best" ranking itself,
//--- not just a side note, so blended accuracy alone can never outrank a directionally-useful era
//--- (see the checkpoint/eta-decay comment in Train()'s era-end block for why that matters).
bool m_bestPassedRecall;
//--- Was the checkpointed era calling BOTH directions? Middle tier of the ranking key - see
//--- isBetterEra. Exists because of a measured failure (2026-08-09, HYBRID era 29): under win-based
//--- scoring an always-Buy model scores EXACTLY chance (P(winLong), ~67% on drifting SP500) while
//--- honest two-sided eras score 63-66% (shorts win less often against the drift), so raw score
//--- ranking crowned the degenerate model, every regression restored back to it, and live NMS
//--- collapsed its constant signal to ~25 trades per era ("barely trades"). bothSidesLive already
//--- blocked it from DEPLOYING via tradeableOK, but among not-yet-deployable eras the score alone
//--- decided - the same phase the coverage credit exists for.
bool m_bestBothSidesLive;
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 00:10:45 -04:00
//--- SLOW-ERA HEARTBEAT (2026-08-10). The era passes are silent by construction - pass 1 logs
//--- nothing and (until tonight) painted nothing for IS bars, pass 2/3 only paint - so when a
//--- restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86s),
//--- the outside view was "Getting ready...", four pegged cores, and an empty log for 20+ minutes,
//--- with the hourly new-bar cache invalidation then cancelling and restarting the unfinished era
//--- forever. Nothing external (thread stacks need a debugger the VPS lacks) can say WHERE the time
//--- goes, so training itself must: cumulative per-era timers around the two candidate costs
//--- (feature-window builds, net forward/backprop) and a heartbeat line that only speaks when an
//--- era is genuinely slow - a healthy run stays exactly as quiet as before.
uint m_eraStartTick;
ulong m_passFeatUs; // cumulative BuildFeatureWindow time this era, microseconds
ulong m_passNetUs; // cumulative feedForward/backProp time this era, microseconds
int m_passHeartbeatPrints;
uint m_lastHeartbeatTick;
//--- How many of pass 1's bars produced a usable feature window, and how many did not. These decide
//--- whether the era does ANY work: add_loop is just "m_passWindowOk > 0", and when it is false
//--- pass 2, pass 3, the era counter and the checkpoint are ALL skipped, so Train() returns having
//--- done nothing and the next call restarts the same era from bar 0 - an infinite, wholly silent
//--- 0->100% scan loop. Counting both sides makes a partial failure (indicators unavailable over
//--- the oldest bars, which is normal and harmless) distinguishable from a total one.
int m_passWindowOk;
int m_passWindowFail;
fix: the recall gate was unsatisfiable and the LR decay was a spiral Both made the run structurally unable to succeed, independently of any signal in the data. Found by reading the 13:01 log. RECALL GATE. m_objectiveMet required Buy, Sell AND Neutral OOS recall each >= 40%. First-touch resolution (ce52654) collapsed Neutral from the ~94% majority it was under exact-pivot labels to a same-bar-tie residue - 250 of 38,261 bars, 0.65% - so the floor was asking the model to identify 40% of coin-flip ties before it could converge. Measured: CONV, LSTM and HYBRID all logged "Neutral:0% (need >=40% each)" on every era. No model could ever satisfy it; every run was destined for the plateau ladder or the era cap. Only the DIRECTIONAL floors are load-bearing for the anti-collapse job the gate exists to do: an all-Neutral model shows Buy and Sell recall at 0% and is blocked by them. Neutral's own floor guarded the mirror bias (over-calling Buy/Sell at Neutral's expense), which was real at 94% prevalence and is not at 0.65% - there, almost never calling Neutral is correct rather than biased. Prevalence-guarded rather than hardcoded off, so it returns by itself if a future label rule makes Neutral substantial again. Deliberately NOT extended to Buy/Sell: exempting a thin directional class reopens the era-44-46 hole, which directionalRecallMeasured only half-covers - it checks those classes were MEASURED, not that they passed. ETA DECAY. A regressing era restored the checkpoint, reset the optimizer and cut eta - all on the FIRST regression. The next era then started from an identical state with a smaller step, regressed again, and got the same treatment. The loop is self-sustaining and cannot discover anything, because rolling the weights back is exactly what removes the exploration that would end it. Measured on PAI: eras 2-11 every one a regression against era 1, eta 0.000594 -> 0.000024, dW/W 0.000%/0.000% from era 2 onward. Ten eras, ~45s each, reproducing era 1 exactly and unable to do anything else. Now requires ETA_DECAY_PATIENCE_ERAS consecutive regressions - the standard ReduceLROnPlateau formulation. A single bad era is noise, and an improving era clears the counter so alternating runs never accumulate into a decay. Build tag -> gate-patience-v3. It had not moved in six commits, which is why the running binary could not be identified from its own log. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:28:58 -04:00
//--- Consecutive regressing eras since the last new best - the patience counter for the checkpoint
//--- restore / eta decay (see ETA_DECAY_PATIENCE_ERAS). Reset by any era that improves.
int m_consecutiveRegressions;
void TrainHeartbeat(const string tag, int done, int total, const string shortLabel);
//--- Progress of the pass currently running, and its name, for the simple panel. Published by each
//--- pass through TrainHeartbeat rather than derived in the UI, because the UI cannot know which
//--- pass owns the counters it can see - deriving it from pass 2's cursor is what made an entire
//--- pass-1 scan display "100%".
int m_passProgressPct;
string m_passLabel;
fix: prebuild and era sized different windows; diag: Train() names its branch TWO things, one incident. 1) THE BUG I SHIPPED IN 0c85c54. m_tuneStartTrainBar is declared, initialised to 0, and NEVER ASSIGNED - the assignment existed before the God-class split and the split dropped it, leaving a dead member. Harmless while nothing read it; a real defect the moment 0c85c54 made StartLabelCachePrebuild() reset dtStudied from it. Train() then computed the window as max(StartTrainBar, floor) while the prebuild computed max(0, floor), where StartTrainBar is the non-zero datetime OnChartEventHandler passes through from the "New Bar" event. The two therefore disagreed about `bars`, so EnsureBarCachesCapacity() saw a changed size at era start, wiped the caches, and re-armed a full 38k-bar prebuild - instead of training. Restored the assignment so both sides evaluate the identical expression. 2) THE REASON IT TOOK ALL NIGHT TO FIND. Train() is a state machine with six early-return branches above the era loop and every one of them is silent. Four charts burned a core each for 15 minutes with an empty journal: the pass heartbeats (694b756) proved the era loop was never reached, no prebuild completion line appeared either, and nothing external can see inside a single MQL5 thread - per-thread CPU says "busy", file writes say nothing, and the VPS has no debugger. That is an undiagnosable state, and it is the thing to fix, not just the bug of the day. ReportTrainStall() now names the branch Train() is taking whenever no era has completed for 3 minutes, at most once a minute per signal, with the state that decides the branch: run/prebuild/simOos/resume flags, era, dtStudied, and - for the cache-invalidation branch specifically - BOTH bar counts, since two sizings disagreeing is exactly what re-arms the prebuild forever. Silent on a healthy run: an era completing resets the clock. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 07:32:08 -04:00
//--- STALL REPORTER. Train() is a state machine with several early-return branches ABOVE the era
//--- loop (OOS simulation walk, label prebuild, history sync, warm-up, cache invalidation), and
//--- every one of them is silent. On 2026-08-10 four charts burned a core each for 15 minutes with
//--- no journal output at all: the pass heartbeats proved the era loop was never reached, and no
//--- prebuild completion line appeared either, so the work was in a branch that cannot say its own
//--- name. Externals (per-thread CPU, file writes) cannot see inside one MQL5 thread, and the VPS
//--- has no debugger - so the state machine has to report itself. m_lastEraCompleteTick is set at
//--- every era end; when Train() is entered and that is stale, ReportTrainStall() names the branch
//--- it is about to take, once per STALL_REPORT_INTERVAL.
uint m_lastEraCompleteTick;
uint m_lastStallReportTick;
void ReportTrainStall(const string branch);
bool m_haveOosCheckpoint;
bool m_oosStable;
bool m_objectiveMet;
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
//--- RAW inputs to the family-wise deployment gate, snapshotted at the same instant as the checkpoint
//--- so the test re-runs on the era that will actually ship rather than on whatever the latest era
//--- happened to score. m_bestBalancedOos alone cannot serve: it is precision already multiplied by
//--- the coverage credit, and the significance test needs the unweighted precision, the chance rate it
//--- is measured against, and the call count that sets its standard error. -1 until the first ranked era.
double m_bestDirPrecPct;
double m_bestChancePrecPct;
int m_bestDirCalls;
fix: NMS gates the TRADE, not just the arrow - one arrow is now one trade NmsLiveAccept() appeared in exactly one place: wrapped around DrawObject(). It never touched dPrevSignal, and dPrevSignal is what LongCondition() / ShortCondition() / SignedAIConfidence() read. So a declustered bar lost its arrow and still opened a position. Measured on SP500 H1 2026-08-09: CONV called a direction on 64% of bars, so the ~500 bars visible on screen held ~320 decisions - and ~40 arrows were drawn. Roughly one arrow per eight positions the EA would take. And the survivors are not a random eighth. Rule 2 of the declustering keeps the HIGHER-CONFIDENCE side of a cluster, so the visible set is systematically the best member of each run. A chart showing the best of every eight decisions and hiding the rest reads far better than the model is - the same best-of-N selection error already corrected in the geometry scan, the indicator tuner, the lag profile and the deploy gate, this time on the display layer, where it is most likely to mislead the person deciding whether to trade. Fixed by neutralising dPrevSignal when NMS rejects, rather than adding a "may trade" flag consulted at each read site: that leaves exactly ONE definition of what the model decided this bar, so the arrow, the panel's "Current signal", the confidence feeding sizing/SL/TP/trailing, the refresh tally and the order itself cannot drift apart again. Also reports the consequence instead of hiding it. Every OOS counter on the era line still scores every directional call - a population ~8x larger than what now trades - so the line carries a second figure: | TRADED (declustered) NN% on N calls (edge +Npp) replaying the identical rule over pass 3 (which walks OOS bars oldest to newest, the same order the live sweep sees). Its cursors are separate members from the live ones so a training pass can never disturb the live chart's declustering. Deliberately NOT switched into selectionScore yet. Declustering cuts coverage from ~64% of bars to ~8%, well under MIN_COVERAGE_FRACTION_OF_BASE_RATE, which would make every checkpoint undeployable overnight - the minRR collision and the recall-floor catch-22 twice over. The floor gets re-derived from these measurements first. Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-09 10:22:31 -04:00
//--- DECLUSTERED OOS tally: the calls that survive NMS, i.e. the ones that actually become positions
//--- now that live NMS gates the trade (see RefreshLatestSignal). Every other OOS counter here scores
//--- a population ~8x larger than the EA trades, so this pair is what "would I have made money" reads.
//--- Era-scoped, reset with the rest of the OOS counters. The three *Nms* cursors are the replay state
//--- for the same rule PruneDirectionalClusters and NmsLiveAccept use - kept separate from the LIVE
//--- cursors so a training pass can never disturb the live chart's declustering.
int m_oosNmsFired;
int m_oosNmsHits;
int m_oosNmsLastBuyIdx;
int m_oosNmsLastSellIdx;
int m_oosNmsKeptIdx;
double m_oosNmsKeptConf;
ENUM_SIGNAL m_oosNmsKeptDir;
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
//--- How many eras the maximum was taken over - the N in the Sidak correction. Counts CANDIDATE eras
//--- (coverage measurable and at least one directional call), not every era, because an era that
//--- called nothing directional was never in the running to become the best and must not inflate N.
//--- Run-scoped: reset with the rest of the best-checkpoint tracking at the top of a fresh run.
int m_deployCandidateEras;
//--- Upper-tail standard normal, Q(z) = P(Z >= z). Abramowitz & Stegun 26.2.17, |error| < 7.5e-8 -
//--- self-contained rather than pulling in MQL5's Math\Stat, which this project has never included and
//--- which drags a chain of headers behind it for the sake of one function.
double NormalUpperTail(double z);
//--- THE GATE. Re-tests the checkpoint that is about to deploy against the null of the MAXIMUM over
//--- m_deployCandidateEras eras, and reports the pieces so the log can show its working. See
//--- DEPLOY_FAMILY_WISE_ALPHA. Returns false (refuse) whenever the inputs are missing.
bool BestCheckpointSurvivesSelection(double &zObs, double &pFamily, int &nTried);
//--- Logs that verdict WITHOUT enforcing it, for the two deploy paths that are explicit operator
//--- decisions (the era cap and the panel's Deploy button). Those stay the operator's call; this just
//--- makes sure the log never lets an authorised deploy read as a validated one.
void ReportSelectionGateVerdict(string context);
//--- Plateau ladder state (see the PLATEAU_* constants). m_erasSinceBestBalanced counts eras since
//--- the last NEW BEST balanced accuracy; m_plateauStage is how far up the escalation it has climbed.
//--- Both are run-scoped and deliberately NOT persisted, matching m_modelEta/m_bestBalancedOos, which
//--- also restart fresh - a resumed run re-earns its patience rather than resuming mid-escalation.
int m_erasSinceBestBalanced;
int m_plateauStage;
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 10:54:09 -04:00
//--- Eras remaining in the current warm-restart boost window (see PLATEAU_RESTART_BOOST): set to
//--- PLATEAU_PATIENCE_ERAS by each boosted restart, decremented by the era-end anneal that walks eta
//--- back to the ceiling, cleared by any new best. Run-scoped and not persisted, same as the ladder
//--- state above.
int m_restartBoostErasLeft;
refactor(ai): nine class-imbalance inputs down to two The imbalance section offered nine controls for one job. Audited against the code, five of them did not do what their names said at the shipped defaults: AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns whenever the adjusted loss is on, which is default. OversampleParity DEAD in training - Training.mqh gated the replay loop on !useLogitAdjustedLoss (correctly, citing Buda et al. 2018). Live only in the online-learning path. EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma damper - "replay minority bars through pass-2 oversampling" was a focal-loss switch. ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25. UseStaticPrior An exact duplicate of FreezePriorCalibration - the two were OR'd together in the single place either is read. So they were not five mechanisms fighting; they were one mechanism plus eight knobs that mostly described machinery that no longer ran. That is worse than a real conflict, because the log agreed with the names: the label-cache line printed "reps up to 28x (90% parity) (seeding era 0's class-balance oversampling)" on every run, describing an oversampling pass that had been switched off. It is fixed here too - it cost this session a wrong diagnosis. The one genuine redundancy was focal loss, running at gamma*0.125 alongside the adjusted loss: two corrections on the same axis, the exact stacking failure this file already cited Buda et al. for in two other places, damped by a replay flag whose replay path was itself dead. Removed rather than re-tuned. The plateau ladder is unaffected - its escape is the learning-rate warm restart; the gamma anneal beside it only ever stepped toward zero. WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze: LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted- Loss boolean, since a strength dial where 0 already means off does not need an on/off switch beside it. FreezePriorCalibration unchanged. It is the only one of the six corrections with a consistency guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection already ranks on - so the loss and the deploy decision optimize one thing. The online continual-learning path keeps its own alpha-balanced focal weight, now as constants pinned to the removed inputs' shipped defaults, so its behaviour is unchanged. It legitimately needs its own correction: ApplyLogitAdjustment() only runs inside a training run, so a deployed model that was reloaded carries no logit offsets and would otherwise stream 31:1 data into itself uncorrected. The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a double fed to a %d conversion and had always emitted a literal 0; the |MR: segment is written as the constant its shipped defaults produced. Dropping either would have re-keyed every model and forced a from-scratch retrain of the one topology currently converged and trading. Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable "neutralized by prior correction" diagnostic. Both builds compile 0 errors, 0 warnings. No retrain forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- m_focalGammaRuntime removed 2026-07-31 with focal loss itself - see the removal note at the
//--- former m_focalGamma above. The plateau ladder keeps its learning-rate warm restart, which was
//--- always the actual escape; the gamma anneal beside it stepped monotonically to zero anyway.
uint m_syncWaitStartTick; // 0 = not waiting on history sync; else GetTickCount() when the wait began
//--- 3 no-op passes on a fresh start (see InitNeuralNetwork()/ResetWeights()), each its own separately-
//--- scheduled Train() call (not a tight in-process loop), so the broker/terminal's history sync gets
//--- several real, wall-clock-separated chances to finish before the era loop commits to a bar count.
int m_warmupPassesRemaining;
//--- fractal/swing-confirmation/trend-context Buy/Sell label cache: the label at a given now-relative
//--- bar index only depends on price/ATR history, never on model state, so recomputing it every era
//--- (as opposed to once per real bar close) is pure waste. Rebuilt fresh whenever `bars` changes
//--- (see the invalidation check in Train()) rather than incrementally appended, since MQL5 timeseries
//--- indices are relative to "now" and shift by one on every new candle - a full rebuild on change
//--- sidesteps needing datetime-keyed/incremental bookkeeping entirely.
diag: is "optimal SL/TP" learnable? Score the features against excursions Proposed direction: train the net to predict entry/SL/TP that maximise return and minimise drawdown, rather than to classify direction. Before rebuilding a head, measure whether the target is learnable at all. That question splits into two that behave nothing alike: HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility clustering is about the most robust regularity in markets. WHICH WAY it goes first (the asymmetry) - direction, which is what every noise-floor verdict in this project has been about. Expectancy comes ONLY from the second. The first buys position sizing and drawdown control - worth having under prop-firm limits, but not an edge: exit management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with expectancy FLAT. Crucially this is NOT already answered. Every MI figure here scored the triple-barrier label, i.e. one specific question at one fixed geometry. A noise-floor result there says nothing about whether excursion MAGNITUDE is learnable - different target, different answer. Four targets, and the verdict is the CONTRAST, printed explicitly because the dangerous misreading of "UP clears" is "we can predict profitable trades": RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that SHOULD clear. Every prior verdict here lacked a control expected to pass; a range target at the floor indicts the measurement, not the market. UP / DOWN - MFE / MAE. ASYMMETRY - up-dn, the only one that can pay. Collected inside the walk the label already does (one max, one min per bar). The early-out when both barriers resolved is GONE: it would have truncated the excursions at whichever bar tripped the last barrier, making the measurement a function of the CURRENT SL/TP - the circularity this is trying to escape. The loop was already bounded by the horizon, so only the average cost moves. Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is fat-tailed and fixed-width bins would put nearly every row in bin 0; it also pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and to the barrier label's ~1.02 instead of confounded by class balance. Two bugs fixed in this code before it ever ran, both of which would have produced a plausible quiet wrong answer rather than an error: - TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the accumulators were reset, so one bar's excursions would be cached under another bar's index. Cleared at the top now, ahead of every return. - An unresolvable bar is still flagged as labelled but carries excursions of exactly 0. Under equal-frequency binning a block of identical zeros drags the lowest cut onto zero and a third of the sample lands in one uninformative bin - a depressed score that reads as "not predictable", a false negative in the direction that would wrongly kill the idea. Rows where both excursions are zero are dropped; price cannot travel zero both ways over a whole horizon. Read-only diagnostic. No topology or label change: no retrain of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 10:22:41 -04:00
//--- Filled in the same pass as the label caches below and gated by the SAME m_labelCacheHasValue, so
//--- a bar either has both or neither and no third validity flag can drift out of step with them.
double m_excUpCache[];
double m_excDownCache[];
feat: first-passage ladder + expectancy scan - price every geometry, not just the chosen one Corrects the premise of the previous plan. Break-even is NOT a ceiling. If the model shifts the win probability on the bars it selects from p0 = m/(m+k) to p0 + d, then EV = (p0+d)*k - (1-p0-d)*m = d*(k+m) because p0*k - (1-p0)*m is zero by construction. The stop:target RATIO is expectancy-neutral - a punishing break-even is exactly repaid by the payoff - and only the real edge d and the TOTAL WIDTH (k+m) move EV. Width matters because the spread is charged once per trade however wide the barriers are, so a narrow barrier spends much of its own range on costs. DeriveBarrierGeometry's own comment already said the ratio buys nothing; the objective just never followed from it. Blocker this had to solve first: m_excUpCache/m_excDownCache hold only MAXIMUM travel each way, and a maximum cannot say which side was reached FIRST - so any geometry other than the walked one was undecidable on precisely the bars where both barriers were touched, ~28% of the sample. - BARRIER_LADDER: per bar, the first-touch AGE for 8 travel distances in each direction, filled during the walk the labels already run. Cursors keep it O(1) amortised per walked bar rather than 16 comparisons. Levels are travel FROM ENTRY, not barrier prices, so one ladder serves both directions and the spread is applied analytically when a level converts back to an SL/TP multiple - storing prices would need four ladders and bake today's spread into the cache. Sized, invalidated and validity-gated with the label caches. - ReportGeometryExpectancyScan: every ladder pair priced exactly off that cache - width in ATR and in SPREADS (cost efficiency, knowable without knowing d), break-even, both base rates, the share of bars resolved inside the horizon, and EV per unit of edge. Compares the widest resolvable pair against the quantile rule's pick. MEASUREMENT ONLY - the quantile rule still chooses. Nothing here can measure d, and width buys nothing if the wider target is less predictable. Base rates are printed beside each break-even because a persistent gap is DRIFT and must not be credited to the model. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 12:59:18 -04:00
//--- First-passage ladder, flat (idx * BARRIER_LADDER_COUNT + level). Value = bars AFTER the entry
//--- bar at which travel first reached BARRIER_LADDER[level] in that direction; 0 = never within the
//--- horizon. Sized and invalidated with the label caches and gated by the same m_labelCacheHasValue,
//--- so a bar has all of it or none. See BARRIER_LADDER for why this exists and why the levels are
//--- travel-from-entry rather than barrier prices.
int m_ladderUpAt[];
int m_ladderDownAt[];
//--- Scratch for the bar TripleBarrierLabel is currently walking, published the same way m_lastExcUp
//--- is and copied into the caches by AdvanceBarrierLabelState under the label's validity flag.
int m_lastLadderUpAt[BARRIER_LADDER_COUNT];
int m_lastLadderDownAt[BARRIER_LADDER_COUNT];
//--- Reports expectancy for every ladder pair - see the definition. Measurement only; it does not
//--- (yet) choose the geometry.
void ReportGeometryExpectancyScan(void);
bool m_labelCacheBuy[];
bool m_labelCacheSell[];
fix: the deploy gate was benchmarking a win rate against a label frequency The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test. That invariant needs reward >= risk, and the measured geometry no longer satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but both-won bars were stripped out of Buy and Sell so the label base rate read 37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma against 37.5% and loses money on every single trade. Live since 217b9bc. Root cause is that label agreement stopped being the same question as trade profitability. Buy implies winLong, but the converse fails on every both-won bar, and the label can only name one of two directions that both pay. So stop asking the model whether it matched a label and start asking whether its trade paid: - cache winLong/winShort per bar beside the label, under the same validity flag; published from the barrier walk before the collapse to 3 classes - dirPrecPct now counts wins on the side actually called - chancePrecPct is max(P(winLong), P(winShort)), MEASURED - the textbook m/(m+k) would credit SP500's drift to the model - the NMS "what would I have made" pair, the live-fired precision, and the IS/OOS cumulative win rates all move to the same test. IS and OOS are read side by side as the overfitting signal, so measuring one in wins and the other in agreement would put a fixed gap between them that has nothing to do with generalization - the confidence threshold is FITTED on wins too, so the operating point maximises what the gate grades - per-class label-agreement precision is still computed and logged; it is the right diagnostic for class separation, just not for a deploy decision - era line renamed dir-precision -> win-rate, chance -> chance=break-even Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 16:00:51 -04:00
//--- Per-bar outcome of each DIRECTION taken on its own, cached beside the label under the same
//--- m_labelCacheHasValue flag. Not derivable from the label: Buy implies winLong, but the converse
//--- fails on every both-won bar, and those are ~27% of the sample under the measured geometry. This
//--- is what the deploy gate scores against - see m_oosWinLongTotal.
bool m_winLongCache[];
bool m_winShortCache[];
bool m_labelCacheHasValue[];
int m_labelCacheBars; // 0 = no cache built yet
datetime m_labelCacheAnchorTime; // m_Time.GetData(0) at last (re)build - 2nd invalidation key
void ComputeLabelForBar(int i, int bars, bool &buy, bool &sell);
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
void AdvanceBarrierLabelState(int i, int bars);
//--- The triple-barrier verdict for one bar - the training TARGET. See BARRIER_TIE_GOES_TO_STOP.
//--- `idx` is a now-relative index; the scan walks FORWARD in time, i.e. toward index 0, and needs
//--- m_barrierHorizonBars of them to exist, so callers must keep idx >= m_barrierHorizonBars.
//--- Returns Neutral for any bar it cannot resolve (no ATR, ran out of history), which is the same
//--- answer as "no setup" and keeps the caller free of a third outcome to handle.
ENUM_SIGNAL TripleBarrierLabel(int idx);
//--- Resolves the SL/TP ATR multiples the label uses from the EA's live SL_Mode/TP_Mode. Split out
//--- because the INTELLIGENT modes scale with AI confidence, which does not exist at label time -
//--- see the definition for why the label uses their zero-confidence base instead.
void BarrierMultiples(double &slMult, double &tpMult);
//--- Median confirmed-ZigZag-leg length over the training window, snapped to the horizon ladder.
int ComputeBarrierHorizonBars(int bars);
//--- Resolves m_barrierHorizonBars exactly once per process, from live buffers. Needed on BOTH paths,
//--- which is the whole reason it is not simply inlined in the prebuild: a DEPLOYED model never enters
//--- Train(), so it never reaches StartLabelCachePrebuild() - yet OnlineLearnStep() reads the horizon
//--- as its confirmation delay. Left unresolved there it would sit at BARRIER_HORIZON_FALLBACK and, on
//--- any symbol whose real horizon is longer, backprop bars whose barriers had not actually resolved -
//--- silent lookahead in the one place that writes to a live, trading model.
void EnsureBarrierHorizon(int bars);
bool m_barrierHorizonResolved;
//--- full per-bar INPUT feature vector cache (everything BufferTempData() computes: ATR-normalized
//--- OHLC, time-of-day encoding, volume delta, AD indicator buffers, ...). Same rationale as the
//--- label cache above - a given now-relative bar index's feature vector only depends on price/
//--- indicator history, never on model state, so recomputing it is pure waste: BufferTempData()
//--- used to rebuild every bar from scratch once per historyBars-wide window it appears in, AND
//--- again on every subsequent era on top of that. Flat array (idx*m_neuronsCount + feature),
//--- not a true 2D array - MQL5 can't dynamically resize an inner dimension. Shares the label
//--- cache's invalidation trigger (see EnsureBarCachesCapacity()) since both are keyed on the exact
//--- same now-relative index frame.
double m_featureCache[];
fix: the trailing incumbent read the future across eras; cold AD blocks cached zeros as truth Three findings from the 2026-08-11 audit: 1. The excursion head's trailing-quantile ring was deliberately never cleared between eras ("a rolling estimate of the market, not of the era") - but pass 3 re-walks the SAME OOS window every era, so at each walk's restart the ring still held the outcome masks of the newest OOS bars from the previous walk: the chronological FUTURE of the bars about to be scored. For the first ~window+horizon pushes of every era the "trailing" incumbent was partly a leading one - conservative for the gate (an informed incumbent is a harder hurdle) but exactly the self-made-artifact class 06d4785 hunts. The ring now clears at era-score reset; the warm-up bars simply don't score the trail race, which the m_excTrailN gating already accounts for. 2. skillTrail compared the head's FULL-block Brier (pro-rated by coverage) against the incumbent's subset sum - valid only if head skill is uniform across the OOS walk, while the trail-scored subset systematically excludes each era's warm-up bars. The audit also found m_excBrierHeadD/BaseD/ m_excOosHitsD declared, zeroed and never accumulated (dead since e2c9593 made every scored bar disjoint). The dead trio is replaced by m_excBrierHeadT: the head's Brier accumulated only on the bars the warm incumbent also scored, so the race now compares both predictors on an identical bar set. 3. The AD/Wyckoff feature blocks read GetData with no EMPTY_VALUE guard; a cold (still-calculating) indicator returns EMPTY_VALUE everywhere, the sanitize loop rewrote that to 0.0, and the bar SUCCEEDED - so BufferTempData cached an all-zero Wyckoff block as a success for the whole bar frame: the one path the f6150ee only-cache-successes rule cannot see, because it never fails (the ba13eef class, arriving through values that never fail; a resumed model's era-0 prebuild starts milliseconds after OnInit). ADIndicatorCold() probes the NEWEST bar - EMPTY_VALUE there means async warm-up (transient reject, retried), while deep bars beyond the buffered depth keep the sanitize loop's neutral-fill so degraded history still trains. Also fixed m_featureCacheValid's declaration comment, which still described the pre-f6150ee cached-miss semantics. Compile: 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 18:20:24 -04:00
bool m_featureCacheHasValue[]; // true once idx has a CACHED SUCCESS (f6150ee: only
// successes are ever cached - a miss is never stored,
// in any form; see BufferTempData's comment)
bool m_featureCacheValid[]; // paired flag, always true when HasValue is true -
// kept for the (currently unreachable) cached-miss
// shape so the cache layout survives f6150ee
fix: a resumed model cached a cold ATR as permanent, so it never trained BufferTempData cached EVERY failure - m_featureCacheHasValue[idx]=true with m_featureCacheValid[idx]=false - and the cache never re-tries a miss. So a single feature read taken before the terminal had finished calculating the indicator buffers marked those bars unusable for the rest of the process, even though the data arrived milliseconds later. MT5 fills an indicator's buffers asynchronously after the handle is created, and a cold ATR returns 0 for EVERY index, not just its warm-up tail. BufferTempDataCompute rejects a bar with no ATR (correctly - the price features would be meaningless), so the whole window failed, and the whole cache was poisoned. Only resumed models were hit, because only they read features that early. Topology.mqh sets m_warmupPassesRemaining = netLoaded ? 0 : 3: a fresh start sits through three separately-scheduled Train() calls before anything touches a feature, which is exactly what those passes are for. A resumed one skips them and TuneIndicatorsAndTrain drives StartLabelCachePrebuild and the MI report from the first chart event. Its rationale - "a restart already has a proven-synced history" - holds for HISTORY and not for INDICATORS, which are recreated every process start. Downstream: BuildFeatureWindow failed on every bar of every era, so add_loop never went true, so pass 2, pass 3, the era counter and the checkpoint were all skipped and pass 1 swept 0->100% forever. The "0 samples" MI report line at startup was the same failure, four seconds earlier, already visible in the log. - a miss is now cached only when it is PERMANENT; the two "not ready yet" guards mark m_featureFailTransient and are recomputed on the next visit. Steady-state cost is ~ind_Periods bars per era, not 54k. - an era that discards itself now drops the feature cache before restarting, so any remaining cause of this state self-heals instead of looping. Deleting the .nnw "fixed" this only by turning the model back into a fresh one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 11:41:31 -04:00
//--- Set by BufferTempDataCompute when it rejected a bar because the data had not ARRIVED yet
//--- (price buffer EMPTY_VALUE, or an ATR the terminal has not finished calculating) as opposed to
//--- the bar being genuinely unusable. BufferTempData reads it to decide whether the miss may be
//--- cached - see both for why a wrongly-cached miss is unrecoverable and what it cost.
bool m_featureFailTransient;
diag: name the cause when every feature window fails, and enforce the width contract Era 0 stalls with "NOT ONE of 54681 scanned bars produced a usable feature window, windows ok=0 failed=54681" and nothing else. That line reads identically for a cold ATR, a conditionally-missing optional feature block and an out-of-range index, so it cannot be diagnosed without one restart per hypothesis. Two changes: 1. WIDTH CONTRACT in BufferTempData. Every enabled block must emit exactly m_neuronsCount values on EVERY bar. A block that emits its values on some bars and skips them on others (indicator, panel or series unavailable for that bar) does not merely shorten the window - it SHIFTS every feature after it into the wrong slot, and the net then trains on silently misaligned inputs that still look like a valid window to everything downstream. Now rejected, rolled back and reported once, naming the optional blocks (XA / SPR / swing context) as the ones carrying an availability test. Worth having independently of the current stall. 2. BuildFeatureWindow records WHICH lookback slot rejected and how much of the window was assembled, and the pass-1 stall report renders it: "slot 0 of 20 REJECTED (window had 0 of 760)" is an indicator warm-up or history-edge read; "every lookback bar ACCEPTED and the window was still short: 640 of 760" is a missing 6-value block. No behaviour change on a healthy run: the width check is an equality that already holds, and the diagnostics render only inside the total-failure branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 10:31:56 -04:00
//--- Why the LAST BuildFeatureWindow failed, so the pass-1 stall report can name a cause instead of
//--- a count. Slot = which lookback position rejected (-1 = none did and the window was still
//--- short); Total = how many values had been assembled when it gave up.
int m_windowFailSlot;
int m_windowFailTotal;
bool m_featureWidthWarned; // one-shot: the width contract is a structural fault
bool BufferTempDataCompute(int idx);
//--- Nearest confirmed (non-repainting) ZigZag pivot at or after fromIdx - see this method's
//--- definition comment and m_useSwingContext's declaration comment for the repainting-embargo
//--- rationale callers must apply to fromIdx before calling this.
bool FindConfirmedZigZagPivot(int fromIdx, int &pivotIdx, double &pivotPrice, bool &pivotIsLow);
bool EnsureBarCachesCapacity(int bars);
//--- Eager label-cache pre-build + true-label tally, run once per fresh start (see
//--- m_warmupPassesRemaining) BEFORE era 0's real training loop begins. Without it, era 0 trains with
//--- m_prevEraTrueBuyCount/Sell/Neutral all still 0, so UpdateClassPriors() has no measured
//--- distribution and era 0 alone gets ZERO correction for label imbalance. Pre-scanning the whole IS
//--- window upfront (reusing the same cache/ComputeLabelForBar() the era loop uses) lets era 0 start
//--- from real measured class base rates.
bool m_labelCachePrebuilt; // true once the one-time pre-scan has completed
bool m_labelPrebuildActive; // true while a chunked pre-scan is in progress
bool m_prebuildSeedPending; // true: era 0's era-start reset must NOT stomp the
// prebuild-seeded m_prevEraTrue* counts with the
// still-empty live tally (see Train()'s era-start block)
int m_labelPrebuildBars;
int m_labelPrebuildOosCutoff;
int m_labelPrebuildIndex;
int m_labelPrebuildBuyCount;
int m_labelPrebuildSellCount;
int m_labelPrebuildNeutralCount;
void StartLabelCachePrebuild(void);
void AdvanceLabelCachePrebuild(void);
//--- Evaluation-only continual-learning OOS simulation: once the core model converges, a CLONE of its
//--- weights (never the production Net itself) walks forward through the OOS window bar-by-bar,
//--- scoring each bar with its current weights THEN learning from it - simulating how the model would
//--- adapt in live/forward trading. This must never feed back into Net or the real OOS convergence
//--- metric (dOosForecast/m_oosSamples), so it's tracked in entirely separate members and the clone is
//--- discarded (never Save()'d) once each walk completes.
CNet *m_simOosNet; // NULL when no simulation is active
bool m_simOosRunActive;
int m_simOosCutoff; // oosCutoff snapshot from the run that converged
int m_simOosBarIndex; // resume point, m_simOosCutoff-1 down to 0
double m_simOosForecast; // smoothed accuracy - separate from dOosForecast
int m_simOosSamples;
void StartOosContinualSimulation(int bars, int oosCutoff);
void AdvanceOosSimulationChunk(void);
//--- same resumability problem one level up: TuneIndicatorsAndTrain()'s own trial loop calls
//--- Train() per trial and used to assume each call ran an entire trial to completion synchronously
int m_tuneTrialIndex; // -1 = no multi-trial tuning run in progress
double m_tuneBestOosForecast;
bool m_tuneLastTrialWasWin;
bool m_tuneHaveBestCheckpoint;
datetime m_tuneStartTrainBar;
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//=== Filter-based indicator auto-tuner (see TuneIndicatorsByFilter) =============================
//--- Replaced a genetic + successive-halving search on 2026-08-01. That search scored every candidate
//--- by TRAINING a throwaway network on it, which cost 1152 eras (9-48 h depending on topology) before
//--- the real model started, and its short screening rungs could not separate candidates at all. The
//--- filter scores candidates by the mutual information between the resulting FEATURES and the LABELS -
//--- arithmetic over the feature cache, no training - so it finishes in seconds and its cost is
//--- independent of topology. See TuneIndicatorsByFilter() for the measurements and the honest limit.
bool m_tuneFilterDone; // the one-shot filter pass has run for this model
//--- Mutual information between one feature column and the 3-class label, and the whole-vector score.
double FeatureColumnMI(const double &vals[], const int &labels[], int n);
//--- Returns the MEAN per-feature marginal MI. Side-effects two more numbers that the mean alone
//--- cannot express, both read by TuneIndicatorsByFilter's report - MQL5 forbids a default value on a
//--- reference parameter, so members rather than out-params:
//--- m_miBestColumn - the STRONGEST single feature's MI. A mean over 26 columns hides one good
//--- column among 25 useless ones, which is exactly the case worth catching.
//--- m_miLabelEntropy - H(Y) in nats for the sampled labels, so MI can be quoted as a FRACTION of
//--- what there is to know. "0.001 nats" means nothing on its own; "0.1% of the
//--- label's entropy" is a magnitude anyone can act on.
fix(autotune): MI scorer read an array nobody filled; add the permutation floor THE TUNER WAS A SILENT NO-OP. Every chart logged auto-tune complete - 17 candidate settings scored in ~139s, feature/label mutual information 0.0000 -> 0.0000 nats (no improvement) 0.0000 is not a weak result, it is a broken measurement: finite-sample MI is biased UPWARD, so even pure noise scores above zero. Cause: ScoreCurrentParamsByMI called BufferTempDataCompute(), which APPENDS the bar's features to TempData and never touches m_featureCache - only the caching wrapper BufferTempData() writes that array. It then read m_featureCache, which ReInitADIndicators had just invalidated. Every column came back constant, FeatureColumnMI returned 0 for all of them, and all 17 candidates tied at exactly zero. 139 s per chart to return the settings it started with. Now reads the values back out of TempData, where they actually land. And an exactly-zero best score is called out as a fault rather than reported as "no improvement", because that is what it is. ADDED: a PERMUTATION BASELINE, which is the diagnostic this project has been missing. MI's finite-sample bias is ~(bins-1)(classes-1)/(2n) nats - at these sample sizes the same order as any real edge in this domain - so a raw MI figure is uninterpretable on its own. Shuffling the labels destroys every genuine association while leaving sample size, binning and class proportions intact, so the score it produces IS this dataset's noise floor, measured rather than approximated. The log now reads feature/label information - X nats against a shuffled-label floor of Y and says outright whether the features carry usable information about the target. It needs no training, no topology and no convergence, so unlike every accuracy number in this codebase it cannot be confounded by an optimizer or an objective. If the score sits on the floor, no change of architecture can help - which is the question the last three days of zero-edge results have been circling. DEPLOY FLOOR: `dirPrecPct > chancePrecPct` passed anything above chance by any amount. At ~11,000 directional calls the standard error of the precision estimate is ~0.4pp, so that gate was accepting sub-one-sigma noise - the perceptron deployed at edge +0pp on 2026-08-01. Now requires EDGE_MIN_SIGMAS (2.0) standard errors above chance, computed from the actual call count, so the bar scales with the evidence instead of needing a hand-picked constant. Recorded with it, because it is why chance is the right reference at all: under a driftless random walk P(touch +k*ATR before -m*ATR) = m/(m+k), and the break-even win rate for a k:m reward:risk trade is ALSO m/(m+k). The label's own base rate IS the break-even rate, at every SL/TP setting. So "beats chance" and "is profitable" are the same test, and no choice of SL/TP can manufacture an edge - only prediction can. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:05:50 -04:00
double ScoreCurrentParamsByMI(bool shuffleLabels = false);
diag(autotune): five permutations was still a coin flip - use a real test The 5-draw z-score shipped an hour ago disproved itself on its first run. All four charts scored the IDENTICAL 0.00401 nats on identical features and identical labels - and reported z of +1.3, +2.0, +4.0 and +4.7. Two "AT THE NOISE FLOOR", two "a real association", same data. The entire swing came from estimating the null's spread from five draws, where the standard deviation of the standard-deviation estimate is ~35%: the denominator was noisier than the effect it was judging. Replaced with an empirical permutation test. 200 draws, p counted by rank with the +1/(B+1) correction (Phipson & Smyth 2010) so p is never reported as exactly zero - no normality assumption and no spread to estimate. The strongest single column is tested against the null distribution OF THE MAXIMUM, which corrects for scoring 26 features at once by construction and is far less conservative than Bonferroni. Affordable because BuildMiSample is now split out of ScoreCurrentParamsByMI and runs ONCE for the whole test - every draw reuses that sample and costs a relabel plus 26 histogram passes, not 2000 feature extractions. The coordinate sweep still calls the combined form, which is correct there: each candidate changes the indicator settings, so its features really do have to be re-extracted. The verdict line keeps both questions apart and prints both answers: the p-value for "is it real", the excess as a percentage of H(Y) for "is it big enough to trade". At n=2000 those can disagree, and collapsing them into one word is how a worthless effect gets called a discovery. Compiles 0 errors / 0 warnings. Build tag permtest-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:45:46 -04:00
//--- The same work split in two, so the permutation test can extract the sample ONCE and reuse it for
//--- every null draw. BuildMiSample returns the sample count (or -1); ScoreMiSample shuffles `labels`
//--- in place when asked, so the observed statistic must always be taken before the first draw.
diag(autotune): a positive control, and a scan that separates "no signal" from "signal knocked out of step" Four architecturally different networks landed on the same precision - Buy 23-25% against a 25.4% base rate, Sell 19-22% against 22.0% - while making completely different calls (HYBRID votes Sell on 69% of bars, PAI on 41%). Precision equal to the base rate is what INDEPENDENCE looks like, and precision under independence is fixed by the label distribution, not by the architecture, so all four converging on it is arithmetic rather than coincidence. Accuracy meanwhile tracks coverage exactly as independence predicts (31.1/30.3/25.0 predicted vs 31.8/28.9/24.6 observed for PAI/CONV/HYB). But "no information in the data" and "information destroyed upstream of every topology" produce that identical picture, and the MI test alone cannot tell them apart either. Two additions: POSITIVE CONTROL. Three "measurements" in this codebase have turned out to be silent no-ops that produced plausible numbers - the MI scorer reading an array nobody filled, the eval-mode guard that switched off the imbalance correction, the alternation gate whose premise was never true. So the estimator now has to prove it responds to a signal known to be present before any floor reading is believed: the label of a neighbouring sample row, ~19 bars away and far inside the 128-bar barrier horizon, so the two outcome windows overlap heavily and MUST be associated. Same binning, same estimator. Near the floor => every MI figure is void. ALIGNMENT SCAN. Re-scores against the label taken from bar i+k for k in -5..+5. A peak at k != 0 is a feature/label misalignment - an off-by-one in the label index, a horizon applied to the wrong bar, a feature window that lags what it claims - which would destroy the information before any topology saw it and would look identical in every accuracy number this EA prints. A flat profile says the features simply do not carry this target. The sampled range is trimmed by |k| at both ends so a shift is measured rather than an edge effect, and both bars must carry a real label. Also: BuildMiSample publishes its stride instead of the report recomputing that arithmetic (it would drift), and the control sizes its buffers from its own sample count rather than the caller's. Compiles 0 errors / 0 warnings, standard and Market. Build tag mi-control-align-v1. Redeploy only - no retrain, no model deletion; the diagnostic runs on resumed models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:12:10 -04:00
//--- labelBarOffset != 0 takes the LABEL from bar i+offset while the features still come from bar i,
//--- which is what the alignment scan needs. The sampled range is trimmed by MiShiftPad() at both
//--- ends - a FIXED amount, never by |offset| - so every build enumerates the same bars in the same
//--- order and two builds can be compared row by row. Returns -1 if the offset exceeds the pad.
diag: is "optimal SL/TP" learnable? Score the features against excursions Proposed direction: train the net to predict entry/SL/TP that maximise return and minimise drawdown, rather than to classify direction. Before rebuilding a head, measure whether the target is learnable at all. That question splits into two that behave nothing alike: HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility clustering is about the most robust regularity in markets. WHICH WAY it goes first (the asymmetry) - direction, which is what every noise-floor verdict in this project has been about. Expectancy comes ONLY from the second. The first buys position sizing and drawdown control - worth having under prop-firm limits, but not an edge: exit management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with expectancy FLAT. Crucially this is NOT already answered. Every MI figure here scored the triple-barrier label, i.e. one specific question at one fixed geometry. A noise-floor result there says nothing about whether excursion MAGNITUDE is learnable - different target, different answer. Four targets, and the verdict is the CONTRAST, printed explicitly because the dangerous misreading of "UP clears" is "we can predict profitable trades": RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that SHOULD clear. Every prior verdict here lacked a control expected to pass; a range target at the floor indicts the measurement, not the market. UP / DOWN - MFE / MAE. ASYMMETRY - up-dn, the only one that can pay. Collected inside the walk the label already does (one max, one min per bar). The early-out when both barriers resolved is GONE: it would have truncated the excursions at whichever bar tripped the last barrier, making the measurement a function of the CURRENT SL/TP - the circularity this is trying to escape. The loop was already bounded by the horizon, so only the average cost moves. Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is fat-tailed and fixed-width bins would put nearly every row in bin 0; it also pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and to the barrier label's ~1.02 instead of confounded by class balance. Two bugs fixed in this code before it ever ran, both of which would have produced a plausible quiet wrong answer rather than an error: - TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the accumulators were reset, so one bar's excursions would be cached under another bar's index. Cleared at the top now, ahead of every return. - An unresolvable bar is still flagged as labelled but carries excursions of exactly 0. Under equal-frequency binning a block of identical zeros drags the lowest cut onto zero and a third of the sample lands in one uninformative bin - a depressed score that reads as "not predictable", a false negative in the direction that would wrongly kill the idea. Rows where both excursions are zero are dropped; price cannot travel zero both ways over a whole horizon. Read-only diagnostic. No topology or label change: no retrain of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 10:22:41 -04:00
//--- `target` selects WHICH outcome the features are scored against (MI_TARGET_*). Anything other
//--- than the barrier class is continuous and is discretised into 3 equal-frequency bins at the end
//--- of the build, so every downstream consumer sees the same 3-class shape it already handles.
diag: MI feature-lag profile - close the blind spot in every MI verdict so far BuildMiSample samples features from ONE bar. So every "MI is at the noise floor" result this codebase has produced - including yesterday's p=0.18 on SP500 H1 - described the ENTRY BAR's 31 features only, while the network is fed 20 bars of them. If information lived at lag 7 and not lag 0, the report would have said "no signal" while the model could still learn. The diagnostic we have been making decisions on had a blind spot exactly the width of the input vector. Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as the existing labelBarOffset and is not interchangeable with it. Shifting the LABEL changes which trade is predicted, so at any non-zero offset the features sit inside the labelled window and the score is lookahead - that is precisely what the alignment scan measures and correctly reports (4.7x more knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label pinned to the entry bar, so every row stays causal. ReportFeatureLagProfile() then scores k = 0..historyBars against the same block-permutation null and reports the deepest lag that clears it - the lookback the data supports, versus the 20 that was picked by hand and never measured. The null is redrawn PER LAG: finite-sample MI bias moves with the realised class counts and bin occupancy, and different rows survive the validity checks at each lag, so one shared floor would be right for lag 0 and wrong everywhere else. Draw count is reduced accordingly (40, not 200) since cost is draws x historyBars; this figure decides a lookback, never a trade. MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that makes two builds comparable row by row. Read-only - no input, topology or label change, so no retrain. Both builds 0/0. Build tag lag-profile-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 13:51:39 -04:00
int BuildMiSample(double &cols[], int &labels[], int labelBarOffset = 0,
diag: is "optimal SL/TP" learnable? Score the features against excursions Proposed direction: train the net to predict entry/SL/TP that maximise return and minimise drawdown, rather than to classify direction. Before rebuilding a head, measure whether the target is learnable at all. That question splits into two that behave nothing alike: HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility clustering is about the most robust regularity in markets. WHICH WAY it goes first (the asymmetry) - direction, which is what every noise-floor verdict in this project has been about. Expectancy comes ONLY from the second. The first buys position sizing and drawdown control - worth having under prop-firm limits, but not an edge: exit management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with expectancy FLAT. Crucially this is NOT already answered. Every MI figure here scored the triple-barrier label, i.e. one specific question at one fixed geometry. A noise-floor result there says nothing about whether excursion MAGNITUDE is learnable - different target, different answer. Four targets, and the verdict is the CONTRAST, printed explicitly because the dangerous misreading of "UP clears" is "we can predict profitable trades": RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that SHOULD clear. Every prior verdict here lacked a control expected to pass; a range target at the floor indicts the measurement, not the market. UP / DOWN - MFE / MAE. ASYMMETRY - up-dn, the only one that can pay. Collected inside the walk the label already does (one max, one min per bar). The early-out when both barriers resolved is GONE: it would have truncated the excursions at whichever bar tripped the last barrier, making the measurement a function of the CURRENT SL/TP - the circularity this is trying to escape. The loop was already bounded by the horizon, so only the average cost moves. Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is fat-tailed and fixed-width bins would put nearly every row in bin 0; it also pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and to the barrier label's ~1.02 instead of confounded by class balance. Two bugs fixed in this code before it ever ran, both of which would have produced a plausible quiet wrong answer rather than an error: - TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the accumulators were reset, so one bar's excursions would be cached under another bar's index. Cleared at the top now, ahead of every return. - An unresolvable bar is still flagged as labelled but carries excursions of exactly 0. Under equal-frequency binning a block of identical zeros drags the lowest cut onto zero and a third of the sample lands in one uninformative bin - a depressed score that reads as "not predictable", a false negative in the direction that would wrongly kill the idea. Rows where both excursions are zero are dropped; price cannot travel zero both ways over a whole horizon. Read-only diagnostic. No topology or label change: no retrain of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 10:22:41 -04:00
int featureBarOffset = 0, int target = MI_TARGET_BARRIER);
//--- Is an "optimal SL/TP" head learnable? Scores the features against excursion magnitude and
//--- asymmetry instead of the barrier class - a different question, see the definition.
void ReportExcursionInformation(void);
feat: derive the ATR multiples from measured excursions - no hardcoded geometry The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in 3482b6c, but the fallback was a hardcoded 2:6 and the geometry scan only ever chose from a hardcoded grid {2,3} x {2,3,4,6,8,10}. Picking the least-bad of eleven guesses is not deriving anything. WHY THE SCAN WAS THE WRONG INSTRUMENT, now measurable rather than argued. It ranks pairings by how predictable their OUTCOME is - a question about direction. The excursion test (2c78f3b) ran on SP500 H1 and direction is the one thing absent: ASYMMETRY p=0.0846, against RANGE/UP/DOWN all at p=0.0050, with RANGE scoring 0.01345 vs a 0.00343 null - 4x, where the barrier label sits at 1.01x. Hence the scan failing its own gate on every run, and its "winner" wandering 2:8 -> 3:8 -> 2:8 -> 2:4 across four runs of the same data. Excursion SIZE is strongly measurable, so derive the geometry from that instead. stop = q25 of measured ADVERSE travel (ordinary noise does not reach it) target = q50 of measured FAVOURABLE travel (reached ~half the time, by construction, inside the horizon) Continuous, in ATR units, superseding the enum multiples. Reachability ("target on X% of bars, stop on Y%") and the implied break-even are printed so the choice is auditable rather than trusted. FIXED-POINT ITERATION, not one-shot. ComputeBarrierHorizonBars scales the horizon with the target (first-passage time grows with the band) and the excursions are measured OVER the horizon, so target -> horizon -> excursions -> target is a real loop - deriving once sizes the target from travel measured under the PREVIOUS horizon. Re-measures until the multiples move <5%, capped at 3 passes, and says so if it does not settle. Does NOT create expectancy, and the log says as much: chance precision equals break-even at every geometry (m/(m+k) on both sides). It buys a target the market reaches and a stop that survives noise. Where Min_Risk_Reward_Ratio forces a target the market rarely reaches, it WARNS rather than overriding - the ratio is the user's risk policy, so the honest move is to state its cost. That is the collision that once rejected 100% of setups. Pinned in the .cfg as doubles appended AFTER this morning's two ints, so .cfg files written earlier today still load (their length guard finds no doubles) and a model that carries them was trained on them and never re-derives. Also fixes a message from e5ceed6 that claimed "this model resumed from disk" unconditionally - it printed above a "seeding era 0" line on a brand-new model, because the branch fires whenever the cache is not built, which is equally true before a fresh model's first prebuild. A diagnostic that misreports its own trigger is worse than one that says nothing: it gets quoted back as evidence. FORCES A FULL RETRAIN (labels change). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 12:06:25 -04:00
//--- Sets the ATR multiples from the measured MFE/MAE quantiles instead of the mode enums. Returns
//--- false (and leaves the configured pair standing) when there are too few resolved excursions.
bool DeriveBarrierGeometry(void);
diag: MI feature-lag profile - close the blind spot in every MI verdict so far BuildMiSample samples features from ONE bar. So every "MI is at the noise floor" result this codebase has produced - including yesterday's p=0.18 on SP500 H1 - described the ENTRY BAR's 31 features only, while the network is fed 20 bars of them. If information lived at lag 7 and not lag 0, the report would have said "no signal" while the model could still learn. The diagnostic we have been making decisions on had a blind spot exactly the width of the input vector. Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as the existing labelBarOffset and is not interchangeable with it. Shifting the LABEL changes which trade is predicted, so at any non-zero offset the features sit inside the labelled window and the score is lookahead - that is precisely what the alignment scan measures and correctly reports (4.7x more knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label pinned to the entry bar, so every row stays causal. ReportFeatureLagProfile() then scores k = 0..historyBars against the same block-permutation null and reports the deepest lag that clears it - the lookback the data supports, versus the 20 that was picked by hand and never measured. The null is redrawn PER LAG: finite-sample MI bias moves with the realised class counts and bin occupancy, and different rows survive the validity checks at each lag, so one shared floor would be right for lag 0 and wrong everywhere else. Draw count is reduced accordingly (40, not 200) since cost is draws x historyBars; this figure decides a lookback, never a trade. MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that makes two builds comparable row by row. Read-only - no input, topology or label change, so no retrain. Both builds 0/0. Build tag lag-profile-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 13:51:39 -04:00
//--- LAG PROFILE: how far back the features still say anything about the entry they precede. Prints
//--- MI at feature lag k = 0..m_historyBars against the same block-permutation null, and returns the
//--- deepest lag that clears it - i.e. the lookback the data actually supports, rather than the 20 that
//--- was picked by hand and never measured.
//--- WHY THIS WAS MISSING AND WHY IT MATTERS: BuildMiSample samples ONE bar. Every "MI is at the noise
//--- floor" verdict this codebase has produced therefore described the ENTRY BAR's features only, while
//--- the network is fed m_historyBars of them. If information lived at lag 7 and not lag 0 the report
//--- would have said "no signal" while the model could still learn - so the diagnostic we have been
//--- deciding on had a blind spot exactly the width of the input vector.
int ReportFeatureLagProfile(void);
//--- Bars trimmed from each end of every MI sample. Must cover the largest offset any caller asks
//--- for: the alignment scan's MI_ALIGN_MAX_SHIFT and the positive control's horizon/4. Expressed
//--- once here so the control cannot drift out of agreement with the range it is sampled over -
//--- which is precisely the failure this replaced.
int MiShiftPad(void) const
{
diag: MI feature-lag profile - close the blind spot in every MI verdict so far BuildMiSample samples features from ONE bar. So every "MI is at the noise floor" result this codebase has produced - including yesterday's p=0.18 on SP500 H1 - described the ENTRY BAR's 31 features only, while the network is fed 20 bars of them. If information lived at lag 7 and not lag 0, the report would have said "no signal" while the model could still learn. The diagnostic we have been making decisions on had a blind spot exactly the width of the input vector. Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as the existing labelBarOffset and is not interchangeable with it. Shifting the LABEL changes which trade is predicted, so at any non-zero offset the features sit inside the labelled window and the score is lookahead - that is precisely what the alignment scan measures and correctly reports (4.7x more knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label pinned to the entry bar, so every row stays causal. ReportFeatureLagProfile() then scores k = 0..historyBars against the same block-permutation null and reports the deepest lag that clears it - the lookback the data supports, versus the 20 that was picked by hand and never measured. The null is redrawn PER LAG: finite-sample MI bias moves with the realised class counts and bin occupancy, and different rows survive the validity checks at each lag, so one shared floor would be right for lag 0 and wrong everywhere else. Draw count is reduced accordingly (40, not 200) since cost is draws x historyBars; this figure decides a lookback, never a trade. MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that makes two builds comparable row by row. Read-only - no input, topology or label change, so no retrain. Both builds 0/0. Build tag lag-profile-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 13:51:39 -04:00
//--- Also covers m_historyBars, because the lag profile shifts the FEATURES that far back and every
//--- build must still enumerate the identical bar set (see BuildMiSample's fixed-pad note - padding
//--- by the requested offset instead is what voided the positive control on 2026-08-02).
return MathMax((int)MathMax(m_historyBars, 0),
MathMax(MI_ALIGN_MAX_SHIFT, MathMax(m_barrierHorizonBars, 1) / 4));
}
diag(autotune): five permutations was still a coin flip - use a real test The 5-draw z-score shipped an hour ago disproved itself on its first run. All four charts scored the IDENTICAL 0.00401 nats on identical features and identical labels - and reported z of +1.3, +2.0, +4.0 and +4.7. Two "AT THE NOISE FLOOR", two "a real association", same data. The entire swing came from estimating the null's spread from five draws, where the standard deviation of the standard-deviation estimate is ~35%: the denominator was noisier than the effect it was judging. Replaced with an empirical permutation test. 200 draws, p counted by rank with the +1/(B+1) correction (Phipson & Smyth 2010) so p is never reported as exactly zero - no normality assumption and no spread to estimate. The strongest single column is tested against the null distribution OF THE MAXIMUM, which corrects for scoring 26 features at once by construction and is far less conservative than Bonferroni. Affordable because BuildMiSample is now split out of ScoreCurrentParamsByMI and runs ONCE for the whole test - every draw reuses that sample and costs a relabel plus 26 histogram passes, not 2000 feature extractions. The coordinate sweep still calls the combined form, which is correct there: each candidate changes the indicator settings, so its features really do have to be re-extracted. The verdict line keeps both questions apart and prints both answers: the p-value for "is it real", the excess as a percentage of H(Y) for "is it big enough to trade". At n=2000 those can disagree, and collapsing them into one word is how a worthless effect gets called a discovery. Compiles 0 errors / 0 warnings. Build tag permtest-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:45:46 -04:00
double ScoreMiSample(const double &cols[], int &labels[], int n, bool shuffleLabels);
//--- The permutation test + verdict, split out of the tuner so it is NOT gated on era 0 with it - see
//--- the definition. Read-only; runs once per attach, whether or not the sweep did.
void ReportFeatureLabelInformation(void);
fix: make the indicator tuner actually measure, and gate what it installs ROOT CAUSE of the zero spread measured on SP500 H1 2026-08-07 (all 17 candidates returned exactly 0.00359 nats): the tune loop re-inits the indicators and then scores, with no RefreshData() between. ReInitADIndicators() does its part - Create() builds a NEW handle carrying the new parameters, and the feature cache is flagged stale so features really are recomputed. But BufferTempDataCompute() reads the CIndicatorBuffer objects, and only Refresh() copies data out of a handle into those. So every candidate was scored on values still held from the PREVIOUS handle. My earlier guess in the diagnostic ("suspect the feature cache") was wrong: the cache invalidation works. Two things land together, because neither is safe alone: 1. RefreshData() after the re-init, so a candidate is scored on its own features. 2. A SELECTION GATE on the install. bestScore is a MAXIMUM over candidates, and the maximum of N draws from a null beats its incumbent almost every time - so "it beat the incumbent" installs noise. This selector is the highest-stakes of the three found in this audit because it ACTS: it overwrites the user's configured indicator settings and forces BuildFreshTopology(), so the network then trains on whatever the noise picked. Fixing (1) without (2) would have made a dormant bug actively harmful. The gate draws the winner's own permutation null once, then corrects the p-value for having chosen it out of N with Sidak: p_family = 1 - (1-p)^N. Sidak rather than the max-of-N resample used by the geometry scan because each candidate here has a DIFFERENT feature set, so their draws cannot be pooled; Sidak needs only the one null. Exact under independence, mildly anti-conservative under positive dependence - stated in the comment rather than hidden. A rejected winner restores the configured settings, which best[] cannot do since the descent mutates it. Also reports the least-ready tunable handle's BarsCalculated(). IndicatorCreate() calculates asynchronously, so if the spread is STILL zero the handles simply are not done and the tuner needs to yield between candidates rather than score them back to back - a state machine like the label prebuild. That distinction is now readable from the log instead of requiring another guess. No input, topology or label change: no retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 09:31:06 -04:00
//--- Smallest BarsCalculated() across the ENABLED tunable indicators, or -1 when none is on. A handle
//--- created by IndicatorCreate() calculates asynchronously, so a candidate scored before its handle
//--- has caught up is scored on an empty or partial buffer. The tuner reports this so "the parameter
//--- change did not reach the features" can be told apart from "it reached them but they weren't ready".
int TunableBarsCalculated(void);
bool m_miReportDone;
//--- Eras the MI report has waited for the cross-asset panel to exist, so it describes the SAME
//--- feature vector training uses. Bounded, so a terminal that never syncs the reference symbols
//--- still gets its diagnostics rather than silently getting none.
int m_miReportDeferrals;
feat(labels): measure which barrier is predictable at entry, don't guess The alignment scan settled the shape of the problem: 4.7x more is knowable 5 bars into a 128-bar window than at the entry the model actually trades. A 6xATR target reached over 128 bars is decided overwhelmingly by what happens DURING the window, so whatever the entry state knows is buried under 128 bars of later noise. That is a property of the TARGET, and it is why four different architectures all landed on precision exactly equal to the base rate - no topology can undo it. So measure the target. For each SL/TP pairing a user can actually select, relabel the same sampled bars and score how much the SAME features say about THAT outcome at entry. Seconds, no training, no topology, and it runs on the diagnostic path that already exists. Ranked on excess over its OWN null as a share of its OWN H(Y), never on raw nats: each geometry has a different class balance, hence a different finite-sample bias and a different amount of information there to find, so raw MI would rank the most BALANCED label rather than the most PREDICTABLE one. The break-even win rate m/(m+k) is printed beside each so the ranking is read next to the bar the model must clear. Stated in the output because it is the easy thing to get wrong: chance precision EQUALS break-even at every geometry, so a tighter target does not hand you expectancy. It buys predictability - less noise piled on top of what the entry state knows - which is the one thing changing topology cannot do. Read-only by construction: it relabels a sampled copy via TripleBarrierLabel(), never writes the label cache (which belongs to the configured geometry), and restores the horizon and overrides it borrowed. The overrides apply only when BOTH are positive, so a half-set pair can never silently relabel a live run. Compiles 0 errors / 0 warnings, standard and Market. Build tag geometry-scan-v1. Redeploy only - no retrain to READ the ranking. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:32:04 -04:00
//--- Ranks every selectable SL/TP pairing by how much the SAME features say about THAT barrier
//--- outcome at ENTRY time - see the definition. Read-only: it relabels a sampled copy, never the
//--- label cache, and restores the barrier state it borrowed.
void ReportBarrierGeometryScan(void);
//--- Scan overrides consulted by BarrierMultiples(). Both > 0 or neither applies; 0 = off. Live only
//--- for the duration of ReportBarrierGeometryScan, and nothing persisted is keyed on them.
double m_barrierScanSlMult;
double m_barrierScanTpMult;
//--- true => BuildMiSample computes each label with TripleBarrierLabel() instead of reading the cache,
//--- because a hypothetical geometry's labels are by definition not cached.
bool m_barrierScanLiveLabels;
fix(labels): the geometry scan rewarded the labels it should reject First run named 3:10 on all four charts, at 2.3x the configured 2:6. That answer was wrong and the fault was the ranking statistic. 3:10 wants a horizon of ~swingMedian*30 (~320 bars) and gets BARRIER_HORIZON_MAX. Clamped, most trades never resolve, the unresolved remainder all lands in Neutral, and H(Y) collapses. The old statistic divided the excess BY H(Y) - so a collapsing denominator made the most degenerate label look like the most predictable one. Every geometry from 2:6 upward was already showing the clamped h128, and the two widest scored highest, which is the fingerprint of the artefact rather than of signal. Two fixes: Rank on the raw excess in nats. Subtracting each geometry's OWN measured null already removes the class-balance bias, which is the only thing the normalisation was ever needed for. Disqualify clamped geometries outright rather than ranking them down. The deployed EA holds until SL or TP with no bar limit, so a truncated label trains the model on a question the strategy never asks. They are still printed, marked '!', so the disqualification is visible instead of a silent omission - and the scan now says so explicitly when nothing eligible is left, because "the limit is the feature set, not the target" is itself the finding in that case. The scan also reports each geometry's directional share and timeout share now. A label nobody can trade is not a candidate however well it scores, and that has to be visible in the same line as the score. Compiles 0 errors / 0 warnings. Build tag geometry-scan-v2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:46:25 -04:00
//--- timed-out labels seen during one geometry's live relabel - see the scan's dir/to columns.
int m_barrierScanTimeouts;
//--- set by ComputeBarrierHorizonBars: this geometry needs MORE time than BARRIER_HORIZON_MAX allows,
//--- so its label truncates a trade the EA would hold to SL/TP. Disqualifies it from the scan.
bool m_barrierHorizonClamped;
double m_miBestColumn;
double m_miLabelEntropy;
diag(autotune): a positive control, and a scan that separates "no signal" from "signal knocked out of step" Four architecturally different networks landed on the same precision - Buy 23-25% against a 25.4% base rate, Sell 19-22% against 22.0% - while making completely different calls (HYBRID votes Sell on 69% of bars, PAI on 41%). Precision equal to the base rate is what INDEPENDENCE looks like, and precision under independence is fixed by the label distribution, not by the architecture, so all four converging on it is arithmetic rather than coincidence. Accuracy meanwhile tracks coverage exactly as independence predicts (31.1/30.3/25.0 predicted vs 31.8/28.9/24.6 observed for PAI/CONV/HYB). But "no information in the data" and "information destroyed upstream of every topology" produce that identical picture, and the MI test alone cannot tell them apart either. Two additions: POSITIVE CONTROL. Three "measurements" in this codebase have turned out to be silent no-ops that produced plausible numbers - the MI scorer reading an array nobody filled, the eval-mode guard that switched off the imbalance correction, the alternation gate whose premise was never true. So the estimator now has to prove it responds to a signal known to be present before any floor reading is believed: the label of a neighbouring sample row, ~19 bars away and far inside the 128-bar barrier horizon, so the two outcome windows overlap heavily and MUST be associated. Same binning, same estimator. Near the floor => every MI figure is void. ALIGNMENT SCAN. Re-scores against the label taken from bar i+k for k in -5..+5. A peak at k != 0 is a feature/label misalignment - an off-by-one in the label index, a horizon applied to the wrong bar, a feature window that lags what it claims - which would destroy the information before any topology saw it and would look identical in every accuracy number this EA prints. A flat profile says the features simply do not carry this target. The sampled range is trimmed by |k| at both ends so a shift is measured rather than an edge effect, and both bars must carry a real label. Also: BuildMiSample publishes its stride instead of the report recomputing that arithmetic (it would drift), and the control sizes its buffers from its own sample count rather than the caller's. Compiles 0 errors / 0 warnings, standard and Market. Build tag mi-control-align-v1. Redeploy only - no retrain, no model deletion; the diagnostic runs on resumed models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 14:12:10 -04:00
//--- bars between two consecutive MI sample rows, set by BuildMiSample - see its note.
int m_miStrideBars;
fix(diag): the symbol sweep was measuring its own sampling, not the market Twelve cells came back with higher-timeframe "signal" 5-9x anything on H1, at p=0.005. It was an artifact, and the sweep's own columns gave it away: excess tracked the sampling STRIDE almost monotonically, and the three D1 cells - stride collapsed to 1-5 bars against a 128-bar horizon, i.e. ~99% window overlap - were the three highest. Three flaws, all the same family: comparing numbers without the spread that belongs to them. 1. THE NULL ASSUMED INDEPENDENCE THE LABELS DO NOT HAVE. Triple-barrier labels overlap; two rows less than one horizon apart share most of their outcome window. A free Fisher-Yates shuffle destroys that dependence along with the association, making the null far narrower than the truth and handing out significance that isn't there - Lopez de Prado ch. 4 arriving through the back door of the significance test. Now permutes contiguous BLOCKS of at least one horizon, so the null keeps the autocorrelation and the p-value means what it says. It degrades honestly: severe overlap leaves few blocks, the null widens, nothing reaches significance. The block count is now printed, because THAT - not the row count - is the sample size a p-value rests on, and a warning fires under 30 blocks so "not significant" is not misread as "no signal" when it means "not enough independent history to tell". 2. THE POSITIVE CONTROL'S STRENGTH DEPENDED ON THE DATASET. It paired each row's label with the NEXT SAMPLE ROW's, whose distance is the stride - so on M5, where stride ran 160-717 bars against a 128-bar horizon, it was pairing two windows that never overlap. All three M5 cells duly reported a FAILED estimator and voided their own results with nothing wrong. A control whose strength varies with the cell cannot certify the cell. Now pinned to a quarter of the horizon, where ~75% overlap is guaranteed by construction. 3. THE LOOKAHEAD VERDICT HAD NO MARGIN. It flagged 7 of 12 cells on gaps of 0.00008-0.00040 nats against a measured null sd of ~0.00030 - noise, every one. Now requires 3 sd, the same discipline the deploy floor applies to precision. Compiles 0 errors / 0 warnings, standard and Market. Build tag blockperm-v1. Supersedes every number from the sweep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 15:11:40 -04:00
//--- independent label blocks the permutation null was built from (rows within one barrier horizon
//--- move together, so THIS - not the row count - is the sample size the p-value really rests on).
int m_miNullBlocks;
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
void TuneIndicatorsByFilter(void);
//================================================================================================
void FinalizeTrainRun(void);
//--- The "this is now THE model" persistence sequence, shared by every deploy path so they can't
//--- drift apart: weights (carrying the current m_trainingComplete flag), the pure-MQL5 inference
//--- self-check, the calibration sidecar, and the EMA shadow. Callers set m_trainingComplete first -
//--- the flag is written INTO the .nnw here. Used by FinalizeTrainRun() (ladder/era-cap/stop deploys)
//--- and by DeployNow()/RetrainDeployed() (the panel buttons).
void PersistDeployedModel(void);
//--- On hitting the per-run era cap: asks the operator whether to keep training (true) or deploy
//--- the best checkpoint and stop (false). Headless (tester/optimizer) can't show a dialog, so it
//--- returns false. See m_maxErasPerRun's declaration comment.
bool PromptContinuePastEraCap(double bestOos);
//--- variables
//--- training control, driven by the control panel (Warrior_EA.mq5); Train()/OnTickHandler
//--- poll these rather than being torn down/rebuilt, so pausing/stopping never loses in-memory state
bool m_trainingPaused; // true: Train() blocks between eras until unpaused
bool m_trainingStopRequested; // true: OnTickHandler stops scheduling new training passes
//--- true for ANY Strategy Tester run - a single backtest AND every optimization pass (MQL_TESTER):
//--- the run must NEVER train. It seeds the agent-local cache from the deployed production model (see
//--- InitNeuralNetwork) and runs pure inference over the window - what a buyer expects from "backtest
//--- my deployed EA", and what makes optimizing TRADING parameters (SL/TP, filters, MM) fast and
//--- comparable (the AI is held fixed, not retrained per config). Training + online continual learning
//--- happen only on a live chart, where this is false.
bool m_inferenceOnly;
//--- true only when the current Net weights came from a saved .nnw on disk, not from a freshly-built
//--- random topology. Used to let an inference-only tester replay a seeded model even if that model's
//--- persisted trainingComplete flag is still false, while still blocking the "no model found, built a
//--- fresh topology" path from placing random-weight trades.
bool m_modelLoadedFromDisk;
feat(ai): real conv receptive field + the reference's channel pool CONV's convolution used window = step = one bar, which is a per-bar projection - a 1x1 conv with a temporal receptive field of ONE BAR. It never mixed information across time, so "convolutional" described the layer type and nothing about what it computed. Same finding that sank HYBRID's LSTM. Pooling was removed on 2026-07-29 for being misconfigured against the conv output's memory layout. That removal was right; leaving the conv at a one-bar window was not. The two belong together: the NeuroNet_DNG reference (references\MQL5\Experts\EDL\Trajectory.mqh layers 2-5, kernels byte-identical to ours) pairs conv(window=2, step=1, window_out=4) with pool(window=4, step=4), and the pool only earns its place because a conv with a real receptive field sits above it. The input is bar-major (BufferTempData appends m_neuronsCount contiguous features per bar), so a flat window of k*m_neuronsCount spans exactly k bars - the receptive field needed NO kernel change. The conv output is position-major, so window == step == window_out is a clean max-over-channels, which is what the reference does and what the existing pool kernels already implement correctly. New chain at H1 defaults (420 = 20 bars x 21): conv1 w=42 s=21 out=8 -> 19 pos x 8 = 152 pool w=8 s=8 -> 19 conv2 w=2 s=1 out=8 -> 18 pos x 8 = 144 (effective field: 3 bars) We deliberately stop before the reference's SECOND pool: a channel pool emits one scalar per position, so a trailing pool would hand the dense stack 18 values and force it to fan out 18 -> 64. That is a bottleneck below every learnable layer - the same class of mistake the 2026-07-29 removal was about. Fixes a latent sizing bug this exposed: CNet's conv/pool position cursor tracked sliding POSITIONS, but a conv's real width is units_count * window_out. Any pool stacked on a conv would therefore have sized against a width window_out times too small and silently built the wrong shape. Both branches now read the built layer's actual Neurons(), which is what the batch-norm branch already did for the same reason. Also closes the architecture-pinning trap: a .nnw persists the window each conv was built with, so an existing CONV/HYBRID model would have loaded cleanly and gone on training under the OLD architecture. The conv weight tensor is (window+1)*window_out, so this cannot be repaired in place - EnforceTopologyContract now detects it, reports both shapes, and retrains. Conv chain shape is derived in one place (ConvReceptiveFieldBars / ConvFirstStagePositions / HasSecondConvStage / ConvOutputPositions / ConvOutputWidth) and consumed by AddConvStage, LstmFanIn and the startup config line, so what is built and what is logged cannot drift. Both builds compile 0 errors, 0 warnings. Forces a CONV and HYBRID retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 20:05:37 -04:00
//--- Set by EnforceTopologyContract() when a just-loaded .nnw was built by a superseded architecture
//--- that cannot be repaired in place (currently: a different conv receptive field, whose weight
//--- tensor is a different SHAPE). InitNeuralNetwork discards the load and retrains. A .nnw persists
//--- the architecture, not just the weights - see AI\Network.mqh CNet::FirstConvWindow.
bool m_topologySuperseded;
//--- true once ValidateCpuInference() has confirmed this model's pure-MQL5 forward pass matches the
//--- compute backend's within tolerance (see CNet::SetCpuInference). Persisted in the .stats sidecar
//--- so an inference-only backtest can run DLL-free; if false (e.g. a conv/LSTM topology not yet
//--- ported, or a validation miss) the backtest falls back to the DLL. Measured at deploy on the
//--- chart (where a backend exists to compare against), never in the tester itself.
bool m_mqlInferenceValidated;
string m_fileName;
string m_folderPath;
//--- which file Train()'s Net.Save() calls (and this method's own Net.Load()) actually target:
//--- the shared FILE_COMMON production weights normally, or a LOCAL per-agent cache file when
//--- running inside the Strategy Tester/optimizer (see InitNeuralNetwork) so that repeated
//--- optimization passes with an unchanged topology can reuse an already-trained model instead of
//--- re-running every era from scratch, without ever touching the live production .nnw/.cfg.
string m_activeFileName;
bool m_activeFileCommon;
//--- Name of the terminal-wide global variable this instance holds as an exclusive claim on
//--- m_activeFileName, or "" when it holds none. See AcquireConfigLock().
string m_configLockName;
//--- user-settable via Inputs.mqh's TrainingOptimizer (SGD or ADAM), read into this member at
//--- construction. Honored by all three signal types - PAI/CONV via BuildFreshTopology(), and
//--- LSTM via CSignalLSTM::AddCustomLayers - since CNeuronLSTMOCL (AI\Network.mqh) now has an
//--- accelerated SGD+momentum path (LSTM_UpdateWeightsMomentum) alongside its original Adam-only
//--- one, and CNet::CNet's GPU/DirectML init gate no longer restricts LSTM topologies to ADAM.
int m_optimizationAlgo;
int m_historyBars;
//--- Input-window derivation for a NEW model (existing models adopt theirs from the .cfg): median
//--- confirmed swing leg from raw highs/lows - strict local extrema over +/-WINDOW_SWING_WING bars,
//--- alternation enforced - snapped down to {12,16,20,24,32}. Raw price, not the ZigZag indicator,
//--- because this runs at init where custom indicators are still cold (see the cold-cache rule).
int DeriveHistoryBars(void);
int m_outputNeuronsCount;
int m_minNeuronsCount;
int m_initialNeuronsCount;
int m_neuronsCount;
double m_neuronsReduction;
int m_hiddenLayersCount;
//--- LSTM-only recurrent hidden-unit count - see LstmHiddenSize's declaration comment
//--- (Variables\Inputs.mqh). Harmless, unused constant contribution to m_fingerprint for MLP/CONV.
int m_lstmHiddenSize;
//--- CONV-only convolutional output-filter count - see ConvFilterCount's declaration comment
//--- (Variables\Inputs.mqh). Harmless, unused constant contribution to m_fingerprint for MLP/LSTM.
int m_convFilterCount;
int m_minTrainYear;
bool m_isInitialized;
fix(deinit): a full model write was running ahead of the cheap cleanup "Abnormal termination" is back, and this time it is not the arrows. The timing names the culprit exactly: 16:02:31.547 OnDeinit: shutting down 16:02:36.003 Abnormal termination <- 4.46 s, MetaTrader gave up 16:02:36.226 chart signals - persisted <- cleanup finished 0.2 s LATE OnDeinit called StopTraining() BEFORE the chart cleanup. StopTraining() finalises an in-flight run, and FinalizeTrainRun() restores the best checkpoint and then persists it - a full ~1MB model write per signal. So the expensive step ran ahead of the cheap bounded one, which is precisely the inversion the shutdown ordering exists to prevent. The previous fix put PersistWeightsOnShutdown last and missed that StopTraining smuggles a second save in at the front. Two changes: Cleanup now runs FIRST, then StopTraining, then the weight save. The visible teardown is cheap and bounded, so it always completes even when everything after it is killed. And the deploy-persist inside FinalizeTrainRun is suppressed during shutdown. RestoreWeights() is an in-MEMORY swap, so the best checkpoint is already the live net by that line, and PersistWeightsOnShutdown writes exactly those weights moments later. The old path wrote the same model twice per signal - eight full writes across four charts - for no benefit. A user-pressed Stop still persists immediately, because nothing else would. Compiles 0 errors / 0 warnings. Build tag deinit-order-v2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 16:06:40 -04:00
//--- true once OnDeinit has begun - see MarkShutdown()/FinalizeTrainRun().
bool m_shutdownInProgress;
int m_fractalPeriods;
//--- "weights" of the 4 CONFIDENCE TIERS a live AI fire can land in (0-100 each), the AI equivalent
//--- of classic indicators' several geometric m_pattern_N members. ConfidenceTier() buckets the
//--- live confidence magnitude (CalibratedConfidenceMagnitude()) into 4 equal bands between the
//--- HEAD'S OWN structural floor and 1.0 - 1/3 for the 3-class softmax, 0.5 for the regression head -
//--- tier 0 = weakest possible directional call, tier 3 = near-certain. UpdateSignalsWeights()
//--- (Expert\ExpertSignalCustom.mqh) then calibrates each tier's weight independently from its OWN
//--- realized win rate, same as any classic pattern.
//---
//--- The defaults span the SAME 0-100 conviction scale the classic patterns use, and that is the
//--- whole mechanism by which one Min vote to open governs both engines. Confidence is expressed as
//--- vote strength rather than as a separate entry floor, so Min_Vote_Open reads directly as "which
//--- tier is good enough", with no second scale to reason about:
//--- 25 tier 0 conf 0.333-0.500 (3-class) / 0.500-0.625 (regression) - a bare plurality
//--- 50 tier 1 conf 0.500-0.667 / 0.625-0.750
//--- 75 tier 2 conf 0.667-0.833 / 0.750-0.875
//--- 100 tier 3 conf 0.833-1.000 / 0.875-1.000 - near-certain
//--- So Min_Vote_Open = 50 lets tier 1 and up trade when the AI votes alone, 75 lets tier 2 and up,
//--- and so on. A near-coin-flip 0.34 argmax is NOT blocked at the AI boundary - it votes 25 and is
//--- filtered by the vote threshold, exactly as a weight-10 classic confirmation would be. Note these
//--- are only the DEFAULTS: with UseDatabaseRanking on, each tier's weight moves to its own measured
//--- win rate, which is precisely why the tiers must be separate patterns rather than one continuous
//--- confidence-to-weight formula - a formula would leave the ranking system nothing to calibrate.
//---
//--- This used to be a single m_pattern_0 covering every fire regardless of confidence. That was a
//--- real bug, not just a missed opportunity: with only one pattern, UpdateSignalsWeights()'
//--- averaging step collapses (average of one value is that value), so the SAME win rate got written
//--- into both the pattern weight AND the module weight (filter.Weight()) - and Direction() multiplies
//--- them together. A genuine 70%-win-rate model was therefore scored as (0.70*70)=49, not 70 - a
//--- quadratic, not linear, derating that got harsher the further from 100% the model's real win rate
//--- was. Classic indicators never hit this because their module weight blends across SEVERAL
//--- differently-performing patterns, diluting any one pattern's own score instead of squaring it.
//--- Splitting into 4 genuinely-different tiers restores that same blending for AI signals.
//---
//--- Defaults (constructor) are graduated 80/87/93/100 - all within the 80-100 band classic
//--- indicators reserve for their most rigorous patterns, since even tier 0 has already passed the
//--- training-completion gate, the confidence floor, and the alternation gate (LongCondition's
//--- comments) before ever reaching here. Hybrid mode (AIType==HYBRID) uses the same shared
//--- weighted-vote path as the other models; no quorum gate is needed anymore.
int m_pattern_0, m_pattern_1, m_pattern_2, m_pattern_3;
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- TRAINING TARGET (Meta_Labeling_Design.md). 0 = per-bar direction (every existing model);
//--- 1 = meta-labeling: one training row per JOURNALED CANDIDATE (a classic pattern instance from
//--- the signal DB), binary label "did the trade this candidate proposes win its triple barrier",
//--- 2-output softmax head. Set once in CSignalMETA's constructor, never mutated afterwards - it
//--- feeds the fingerprint (|TGT:META token) like any other identity-defining member.
int m_trainTarget;
//--- Meta candidate store for the CURRENT era's bar grid, populated by MetaPrepareEra() (overridden
//--- in CSignalMETA; empty and unused for direction models). MQL5 series indices shift on every new
//--- closed bar, so these are re-resolved from the corpus' fixed bar TIMES at each era start -
//--- same invalidation reasoning as the label caches.
int m_metaCandBar[]; // series index of the candidate's fire bar
char m_metaCandSide[]; // +1 long / -1 short
double m_metaCandNetVote[]; // the firing filter's own vote margin (raw weight units)
short m_metaCandFamily[]; // 0=MA 1=RSI 2=MACD 3=Ichimoku
short m_metaCandPattern[]; // Pattern_N within the family
int m_metaCandCount;
//--- per-bar chain: m_metaCandHead[barIdx] -> first candidate id at that bar (-1 none),
//--- m_metaCandNext[candId] -> next candidate at the same bar. Rebuilt with the store.
int m_metaCandHead[];
int m_metaCandNext[];
//--- Candidate id for each pass-2 queue slot, parallel to m_isTrainQueue (see Training.mqh's
//--- queueing block); -1 on every slot for direction models. Swapped in lockstep by the shuffle.
int m_isTrainQueueCand[];
//--- Per-family (0-3) and per-side (0=long 1=short) OOS decomposition of the meta head's era -
//--- candidates / base wins / operating-point trades / wins among trades. The aggregate META line
//--- can hide a deployable subset inside a blended 66%: measured 2026-08-13, 350 eras of real
//--- +1-2pp ranking skill never cleared a 67.5% break-even IN AGGREGATE, and whether any family
//--- or side clears it alone is exactly what this answers. Reset each era beside m_oosBuyFired.
int m_metaFamCand[4], m_metaFamWins[4], m_metaFamFired[4], m_metaFamFiredWins[4];
int m_metaSideCand[2], m_metaSideWins[2], m_metaSideFired[2], m_metaSideFiredWins[2];
//--- functions
virtual bool InitIndicators(CIndicators *indicators);
//--- sets ID/m_id/m_folderPath/m_fileName/m_pattern_count from the subclass constructor - defaults
//--- to 4 (the confidence tiers - see m_pattern_0's declaration comment), not 1
void SetIdentity(string id, string shortId, int patternCount = 4);
//--- hook for neuron-type-specific layers (Conv+Pool, LSTM, ...); default is a plain perceptron (no-op)
virtual bool AddCustomLayers(CArrayObj *topology) { return true; }
refactor: compose topologies from named stages; drop dead code DRY - topology construction --------------------------- CSignalCONV and CSignalHYBRID each built the Conv+Pool front-end from scratch; CSignalLSTM and CSignalHYBRID each built the LSTM stage from scratch. The duplicates had already drifted: HYBRID guarded the LSTM step with MathMax(1, historyBars/2), CSignalLSTM divided unguarded, so a historyBars of 1 gave two different steps for what is documented as the same layer. Extracted AddConvPoolStage() and AddLstmStage() onto CExpertSignalAIBase. The three overrides are now compositions: CONV = AddConvPoolStage LSTM = AddLstmStage HYBRID = AddConvPoolStage && AddLstmStage HYBRID's "matches the standalone CONV front-end exactly, then adds LSTM" is enforced by construction instead of by comment. Took the guarded step for both. Also fixed a descriptor leak the duplicates shared: on a failed topology.Add() the CLayerDescription was neither owned by the array nor deleted. Dead code --------- - CNet::SaveCheckpoint / CNet::LoadCheckpoint (123 lines). Superseded by the in-memory CaptureWeights/RestoreWeights pair; Network.mqh:1312 already said so ("This replaces the file-based SaveCheckpoint/LoadCheckpoint"). Zero call sites - every remaining mention was a comment. The five comments that referenced them have been reworded rather than left dangling. - CExpertSignalCustom::CheckForDuplicateTrade / FindLastTradeIndex / UpdateTradeStatusAndExit: declared, never defined anywhere, never called. They only made it look as though duplicate-trade detection existed. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:38:05 -04:00
//--- Reusable front-end stages, composed by the AddCustomLayers() overrides. Each subclass names the
//--- stages it wants instead of re-declaring the CLayerDescription fields, so the topologies cannot
fix(ai): drop the conv pooling stage - it reduced across filters, not time FeedForwardConv emits POSITION-MAJOR output, matrix_o[out + window_out * i], so one bar's window_out filter responses are contiguous and consecutive bars sit window_out apart. Both pooling implementations (FeedForwardProof and CPU_FeedForwardProof) slide FLAT over that buffer - pos = i * step, reducing `window` CONSECUTIVE elements. On a position-major layout those neighbours are different FILTERS of the same bar, never one filter across time. At the shipped 3/2 the pool computed max(bar0_f0, bar0_f1, bar0_f2), then max(bar0_f2, bar0_f3, bar0_f4), with every 8th window straddling a bar boundary. So it collapsed unrelated feature detectors into whichever fired hardest, passed gradient to that winner only, and halved the feature map while doing it - all below every learnable layer, where nothing above can recover it. The removed inputs' own labels ("3 Bars") show time-axis pooling was the intent throughout. Measured cost: CONV sat pinned at ~40% balanced accuracy for 510 eras with Sell recall 0%, while plain MLPs on the same data reached 57-61%. HYBRID, which also carried this stage, came second-worst of the batch-norm group. Not fixable in the topology: pooling one filter across time needs a stride of window_out BETWEEN samples within a window, which a consecutive-window kernel cannot express at any window/step. That needs a stride-aware kernel in Network.cl + WarriorCPU.cpp + WarriorDML.cpp and a DLL rebuild, and is only worth doing if a conv front-end earns its place without downsampling first - with 20 sliding positions there is little to gain by halving them. ConvPoolWindow/ConvPoolStep and their enums are removed with it, along with the |CP: fingerprint term added earlier today. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:28:44 -04:00
//--- silently drift apart: HYBRID is *defined* as AddConvStage + AddLstmStage, which makes its
refactor: compose topologies from named stages; drop dead code DRY - topology construction --------------------------- CSignalCONV and CSignalHYBRID each built the Conv+Pool front-end from scratch; CSignalLSTM and CSignalHYBRID each built the LSTM stage from scratch. The duplicates had already drifted: HYBRID guarded the LSTM step with MathMax(1, historyBars/2), CSignalLSTM divided unguarded, so a historyBars of 1 gave two different steps for what is documented as the same layer. Extracted AddConvPoolStage() and AddLstmStage() onto CExpertSignalAIBase. The three overrides are now compositions: CONV = AddConvPoolStage LSTM = AddLstmStage HYBRID = AddConvPoolStage && AddLstmStage HYBRID's "matches the standalone CONV front-end exactly, then adds LSTM" is enforced by construction instead of by comment. Took the guarded step for both. Also fixed a descriptor leak the duplicates shared: on a failed topology.Add() the CLayerDescription was neither owned by the array nor deleted. Dead code --------- - CNet::SaveCheckpoint / CNet::LoadCheckpoint (123 lines). Superseded by the in-memory CaptureWeights/RestoreWeights pair; Network.mqh:1312 already said so ("This replaces the file-based SaveCheckpoint/LoadCheckpoint"). Zero call sites - every remaining mention was a comment. The five comments that referenced them have been reworded rather than left dangling. - CExpertSignalCustom::CheckForDuplicateTrade / FindLastTradeIndex / UpdateTradeStatusAndExit: declared, never defined anywhere, never called. They only made it look as though duplicate-trade detection existed. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:38:05 -04:00
//--- "matches the standalone CONV front-end exactly, then adds LSTM" contract structural rather than
//--- a comment. (They had already drifted - HYBRID guarded the LSTM step with MathMax(1,...) and
//--- CSignalLSTM did not, so a historyBars of 1 gave the two a different step.)
fix(ai): drop the conv pooling stage - it reduced across filters, not time FeedForwardConv emits POSITION-MAJOR output, matrix_o[out + window_out * i], so one bar's window_out filter responses are contiguous and consecutive bars sit window_out apart. Both pooling implementations (FeedForwardProof and CPU_FeedForwardProof) slide FLAT over that buffer - pos = i * step, reducing `window` CONSECUTIVE elements. On a position-major layout those neighbours are different FILTERS of the same bar, never one filter across time. At the shipped 3/2 the pool computed max(bar0_f0, bar0_f1, bar0_f2), then max(bar0_f2, bar0_f3, bar0_f4), with every 8th window straddling a bar boundary. So it collapsed unrelated feature detectors into whichever fired hardest, passed gradient to that winner only, and halved the feature map while doing it - all below every learnable layer, where nothing above can recover it. The removed inputs' own labels ("3 Bars") show time-axis pooling was the intent throughout. Measured cost: CONV sat pinned at ~40% balanced accuracy for 510 eras with Sell recall 0%, while plain MLPs on the same data reached 57-61%. HYBRID, which also carried this stage, came second-worst of the batch-norm group. Not fixable in the topology: pooling one filter across time needs a stride of window_out BETWEEN samples within a window, which a consecutive-window kernel cannot express at any window/step. That needs a stride-aware kernel in Network.cl + WarriorCPU.cpp + WarriorDML.cpp and a DLL rebuild, and is only worth doing if a conv front-end earns its place without downsampling first - with 20 sliding positions there is little to gain by halving them. ConvPoolWindow/ConvPoolStep and their enums are removed with it, along with the |CP: fingerprint term added earlier today. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:28:44 -04:00
bool AddConvStage(CArrayObj *topology);
refactor: compose topologies from named stages; drop dead code DRY - topology construction --------------------------- CSignalCONV and CSignalHYBRID each built the Conv+Pool front-end from scratch; CSignalLSTM and CSignalHYBRID each built the LSTM stage from scratch. The duplicates had already drifted: HYBRID guarded the LSTM step with MathMax(1, historyBars/2), CSignalLSTM divided unguarded, so a historyBars of 1 gave two different steps for what is documented as the same layer. Extracted AddConvPoolStage() and AddLstmStage() onto CExpertSignalAIBase. The three overrides are now compositions: CONV = AddConvPoolStage LSTM = AddLstmStage HYBRID = AddConvPoolStage && AddLstmStage HYBRID's "matches the standalone CONV front-end exactly, then adds LSTM" is enforced by construction instead of by comment. Took the guarded step for both. Also fixed a descriptor leak the duplicates shared: on a failed topology.Add() the CLayerDescription was neither owned by the array nor deleted. Dead code --------- - CNet::SaveCheckpoint / CNet::LoadCheckpoint (123 lines). Superseded by the in-memory CaptureWeights/RestoreWeights pair; Network.mqh:1312 already said so ("This replaces the file-based SaveCheckpoint/LoadCheckpoint"). Zero call sites - every remaining mention was a comment. The five comments that referenced them have been reworded rather than left dangling. - CExpertSignalCustom::CheckForDuplicateTrade / FindLastTradeIndex / UpdateTradeStatusAndExit: declared, never defined anywhere, never called. They only made it look as though duplicate-trade detection existed. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:38:05 -04:00
bool AddLstmStage(CArrayObj *topology);
//--- Which front-end stages this subclass's AddCustomLayers() actually appends. Declared once per
//--- subclass and consumed by everything that has to reason about the built shape (the LSTM capacity
//--- budget, the startup config line), so those can never disagree with what was constructed. A
//--- virtual rather than an AIType check, so a future composition cannot silently get the wrong answer.
virtual bool UsesConvStage(void) const { return false; }
virtual bool UsesLstmStage(void) const { return false; }
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-13 06:52:31 -04:00
//--- META-TARGET SEAMS (all no-ops for direction models; overridden only by CSignalMETA).
bool IsMetaTarget(void) const { return m_trainTarget == 1; }
//--- Resolve the candidate corpus onto this era's bar grid (fills the m_metaCand* store). Called at
//--- every era start, right after the bar grid is sized; returning false aborts the training run.
virtual bool MetaPrepareEra(const int bars) { return true; }
//--- Append the per-candidate setup descriptor to TempData, AFTER BuildFeatureWindow() has filled
//--- the shared bar window. The input layer is sized historyBars*features + MetaDescWidth(), so
//--- every feedForward on a meta net MUST run this between window build and forward.
virtual void AppendCandidateFeatures(const int candId) {}
//--- Width of that descriptor; 0 for direction models so NetInputWidth() stays byte-identical.
virtual int MetaDescWidth(void) const { return 0; }
//--- The one true input width every feedForward guard compares against.
int NetInputWidth(void) const { return (int)m_historyBars * m_neuronsCount + MetaDescWidth(); }
//--- P(win) from the 2-output head's raw activations in TempData (after Net.getResults) - the
//--- 2-class softmax collapses to a logistic over the logit difference. Same CLASS_LOGIT_SCALE the
//--- training gradient applies, so the probability is the one the loss was optimizing. -1 = no data.
double MetaWinProbability(void)
{
if(TempData.Total() < 2)
return -1.0;
double z = CLASS_LOGIT_SCALE * (TempData.At(0) - TempData.At(1));
return 1.0 / (1.0 + MathExp(-z));
}
//--- Triple-barrier outcome of the candidate's own side at its fire bar - the meta LABEL. Reads the
//--- side-conditional win caches the label prebuild already computes for every bar; loss AND
//--- timeout are both 0, matching the design ("win=1 / loss-or-timeout=0").
bool MetaCandidateWon(const int candId, const int barIdx)
{
if(barIdx < 0 || barIdx >= ArraySize(m_labelCacheHasValue) || !m_labelCacheHasValue[barIdx])
return false;
if(m_metaCandSide[candId] > 0)
return (barIdx < ArraySize(m_winLongCache)) ? m_winLongCache[barIdx] : false;
return (barIdx < ArraySize(m_winShortCache)) ? m_winShortCache[barIdx] : false;
}
//--- First candidate id at a bar (-1 none) / next in the same-bar chain.
int MetaCandFirst(const int barIdx) const
{ return (barIdx >= 0 && barIdx < ArraySize(m_metaCandHead)) ? m_metaCandHead[barIdx] : -1; }
int MetaCandNext(const int candId) const
{ return (candId >= 0 && candId < ArraySize(m_metaCandNext)) ? m_metaCandNext[candId] : -1; }
//--- AddConvStage runs BEFORE AddLstmStage wherever both are present (HYBRID), so the LSTM is fed the
//--- conv feature map rather than the raw flattened input.
bool HasConvBeforeLstm(void) const { return UsesConvStage() && UsesLstmStage(); }
feat(ai): real conv receptive field + the reference's channel pool CONV's convolution used window = step = one bar, which is a per-bar projection - a 1x1 conv with a temporal receptive field of ONE BAR. It never mixed information across time, so "convolutional" described the layer type and nothing about what it computed. Same finding that sank HYBRID's LSTM. Pooling was removed on 2026-07-29 for being misconfigured against the conv output's memory layout. That removal was right; leaving the conv at a one-bar window was not. The two belong together: the NeuroNet_DNG reference (references\MQL5\Experts\EDL\Trajectory.mqh layers 2-5, kernels byte-identical to ours) pairs conv(window=2, step=1, window_out=4) with pool(window=4, step=4), and the pool only earns its place because a conv with a real receptive field sits above it. The input is bar-major (BufferTempData appends m_neuronsCount contiguous features per bar), so a flat window of k*m_neuronsCount spans exactly k bars - the receptive field needed NO kernel change. The conv output is position-major, so window == step == window_out is a clean max-over-channels, which is what the reference does and what the existing pool kernels already implement correctly. New chain at H1 defaults (420 = 20 bars x 21): conv1 w=42 s=21 out=8 -> 19 pos x 8 = 152 pool w=8 s=8 -> 19 conv2 w=2 s=1 out=8 -> 18 pos x 8 = 144 (effective field: 3 bars) We deliberately stop before the reference's SECOND pool: a channel pool emits one scalar per position, so a trailing pool would hand the dense stack 18 values and force it to fan out 18 -> 64. That is a bottleneck below every learnable layer - the same class of mistake the 2026-07-29 removal was about. Fixes a latent sizing bug this exposed: CNet's conv/pool position cursor tracked sliding POSITIONS, but a conv's real width is units_count * window_out. Any pool stacked on a conv would therefore have sized against a width window_out times too small and silently built the wrong shape. Both branches now read the built layer's actual Neurons(), which is what the batch-norm branch already did for the same reason. Also closes the architecture-pinning trap: a .nnw persists the window each conv was built with, so an existing CONV/HYBRID model would have loaded cleanly and gone on training under the OLD architecture. The conv weight tensor is (window+1)*window_out, so this cannot be repaired in place - EnforceTopologyContract now detects it, reports both shapes, and retrains. Conv chain shape is derived in one place (ConvReceptiveFieldBars / ConvFirstStagePositions / HasSecondConvStage / ConvOutputPositions / ConvOutputWidth) and consumed by AddConvStage, LstmFanIn and the startup config line, so what is built and what is logged cannot drift. Both builds compile 0 errors, 0 warnings. Forces a CONV and HYBRID retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 20:05:37 -04:00
//--- Conv chain shape - see the definitions above AddConvStage. Every consumer reads these rather
//--- than re-deriving the arithmetic, so the built topology and the logged shape cannot disagree.
int ConvReceptiveFieldBars(void) const;
int ConvFirstStagePositions(void) const;
bool HasSecondConvStage(void) const;
int ConvOutputPositions(void) const;
int ConvOutputWidth(void) const;
fix(ai): stop the shutdown save from resurrecting reset weights; size HYBRID's LSTM to its real fan-in ResetWeights already deletes the whole model set - .nnw, .cfg, _ckpt.tmp, .stats, _shadow.nnw - and clears both the .arrows sidecar and the drawn chart objects. What undid it was PersistWeightsOnShutdown: detaching the EA after a reset but before an era completed re-created a .nnw from the freshly-built, never-run net, so the next attach loaded an era-0 stub instead of starting clean. For LSTM/HYBRID that stub is worse than nothing - a layer that has never run a forward pass has m_iInputs<=0, so Save omits every LSTM buffer (see 413ff7e). Skip the save when no era completed and no model was loaded; that is exactly the post-reset and first-attach state. Also sweep _shadowclone.tmp, which the reset did not cover. Separately, ComputeLstmHiddenSize budgeted every topology against the flattened input (historyBars x neuronsCount). True for LSTM, wrong for HYBRID, where AddConvStage runs first and the LSTM is fed the conv feature map - historyBars x convFilterCount, 160 rather than 420 at H1 defaults. The quadratic is dominated by the inputs term, so overstating the fan-in 2.6x cost a full ladder step (16 units where the budget affords 32). New virtual HasConvBeforeLstm() feeds LstmFanIn(), so composition decides this rather than an AIType check. desc.window is advisory only - CNet never passes it to the layer - but is now truthful for the same reason. Derived values stay out of the weights-filename fingerprint and are adopted from the .cfg, so existing models keep their saved width; only fresh ones pick up the corrected budget. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 13:06:09 -04:00
//--- Actual input width the LSTM block sees, which is NOT always the flattened input.
int LstmFanIn(void) const;
//--- " | conv 21->8 x20 bars | lstm 160->32" for the startup config line; "" when neither applies.
string FrontEndConfigSummary(void) const;
feat(ai): batch normalization between dense layers The only bounded stage in the entire forward path was the sigmoid classification head - every hidden stage is PRELU. That is a network with no internal scale control, and the failure ordered exactly by depth: on SP500 H1 the shallow perceptron held ~52% balanced accuracy while the deepest topology sat on the 33.3% one-class floor, with the per-bar logit spread decaying monotonically (0.45 -> 0.38 over ~200 eras) until the evidence tilt fell under the class-prior tilt. That is the signature of internal covariate shift, which chapter 6.1 of the reference book is entirely about and which the NeuroNet_DNG engine addresses with a layer this project never had. Two mechanisms make this the right fix rather than more hyperparameter nudging: - it decouples WEIGHT_DECAY from the learned function (van Laarhoven 2017) - with a normalized layer downstream, decay can no longer grind the discriminative signal away, it only rescales the effective learning rate; - it is the precondition for ever running an unbounded logit head here. The 2026-07-27 attempt blew up (IS error 5.6e15) precisely because nothing upstream constrained scale. Implementation notes: - CNeuronBatchNormOCL computes host-side rather than as a fourth copy of a kernel across Network.cl + WarriorCPU.cpp + WarriorDML.cpp. The math is elementwise O(n); this way it behaves identically on all four compute tiers, needs no DLL rebuild, and cannot drift between backends. Same precedent as the softmax+CCE gradient and the per-sample loss weighting, both computed in MQL5 for that reason. - Statistics are exponential moving, not a stored mini-batch: training is pure online SGD, one update per sample, so there is no batch to average over. BatchNormWindow is an EMA window length. - gamma/beta are excluded from weight decay, deliberately - decaying gamma toward zero is the exact pathology being fixed. - The layer self-sizes from whatever sits below it, because a conv/pool stage's output width is derived inside the CNet constructor and is not knowable to the topology builder. - Checkpoint capture/restore/blend carry gamma/beta and the running statistics alongside the dense matrix, so the plateau ladder cannot restore a mismatched pair. - SeedOutputLayerBias accepted only an exact defNeuronBaseOCL as the weight-carrying penultimate layer; with normalization enabled that is the batch-norm layer, so the cold-start bias seed would have silently stopped being applied. - Refuses to build, loudly, if a topology asks for normalization with no compute backend at all - rather than quietly training a different architecture than the one requested. EnableBatchNorm (default on) and BatchNormWindow (1000 samples) are inputs so the effect can be A/B'd without a recompile. Both feed the weights-filename fingerprint, appended conditionally so existing non-BN configs keep their fingerprints and are not forced to retrain. Verified: analytic gradients match finite differences to 1.5e-7 relative over 200 random cases; a faithful port of the full forward/backward chain collapses to the 33.3% floor by era 4 without this layer and holds 36-43% with it. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 12:34:29 -04:00
//--- Appends a batch-normalization layer, or does nothing (returning success) when EnableBatchNorm is
//--- off. `units` is advisory only - CNet sizes the layer from whatever sits below it, because a conv
//--- or pool stage's output width is derived inside the CNet constructor and is not knowable here.
//--- See AI\NeuronBatchNorm.mqh for what the layer does and why it exists.
bool AddBatchNormStage(CArrayObj *topology, int units);
//--- hardcoded activation for the common tapering Dense hidden-layer stack built by
//--- BuildFreshTopology() (below AddCustomLayers, above the output layer). PRELU (leaky ReLU,
//--- 0.01 slope) is the default: it doesn't saturate/vanish the way TANH does across a deep
//--- taper, and is what the Conv layer itself already uses (see CSignalCONV::AddCustomLayers).
//--- CSignalLSTM overrides this to TANH instead - these Dense layers sit directly on top of the
//--- LSTM layer's own TANH-bounded ([-1,1]) output, so keeping them bounded too avoids feeding an
//--- unbounded activation straight off a bounded recurrent output, which is untested territory
//--- here and not what the user asked for ("Tanh for LSTM gates").
virtual ENUM_ACTIVATION HiddenLayerActivation(void) { return PRELU; }
fix: stop a .nnw from pinning a superseded architecture A .nnw persists the ARCHITECTURE, not just the weights: Save writes (int)activation per neuron and Load reads it straight back. The activation chosen in BuildFreshTopology() therefore only ever reached a brand-new topology - every reload restored the file's value and the next save wrote it back out, so a wrong value could never heal while the source read as though it were already fixed. That is how five models kept training with an unbounded NONE classification head for a full day after the 07-28 revert to SIGMOID. Confirmed by parsing the binaries: 848cb42c.nnw / 2e754b43.nnw carry `act=NONE` on the 3-neuron output layer, while a genuinely reset model of the same config carries act=SIGMOID. In the log it showed as negative "OOS raw out" values - impossible under sigmoid - escalating to a 4.14e13 logit spread with all three classes numerically identical (input-independent output) and balanced accuracy pinned on the 33.3% one-class floor. - OutputLayerActivation() is now the single source of truth, called by both BuildFreshTopology() and the new load-time repair, so the two can no longer diverge the way a duplicated literal did. - CNet::EnforceOutputActivation() re-asserts it after Load and reports the stale value; CExpertSignalAIBase::EnforceTopologyContract() logs the repair loudly, since weights learned under the old head may not be worth keeping even once the head is corrected. - Hidden layers are deliberately left alone: they legitimately differ per stage (PRELU dense/conv, NONE pool, TANH LSTM). Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 12:00:40 -04:00
//--- Single source of truth for the output head's activation. BuildFreshTopology() stamps it into a
//--- NEW topology; EnforceTopologyContract() re-asserts it after every Load(), because a .nnw
//--- persists the activation and would otherwise pin a superseded architecture forever (see
//--- CNet::EnforceOutputActivation's declaration comment in AI\Network.mqh for the incident this
//--- comes from). Deliberately one expression called from both places - when these were two separate
//--- literals, changing the head in BuildFreshTopology() silently did nothing to any existing model.
//--- Regression (1 output): TANH - its native [-1,1] range maps directly onto the -1/0/1
//--- Sell/Neutral/Buy target convention. Classification (3 outputs): SIGMOID - see the long
//--- rationale at BuildFreshTopology()'s use of this method for why the head must stay BOUNDED.
ENUM_ACTIVATION OutputLayerActivation(void) const { return (m_outputNeuronsCount == 1) ? TANH : SIGMOID; }
refactor(ai): derive the first dense layer's width instead of asking for it InitialNeurons was an input whose only defensible value depends on two things the user cannot see when picking from a dropdown: how wide the input vector ended up after feature selection, and how much in-sample data the study period actually yields. Left to a hand-picked constant it was badly wrong - 500 units against a 420-wide input is 210,500 weights, 72% of a 292,583-weight model, against ~36,500 training bars of which only ~2,236 are directional. That is 6.6 weights per training bar, and it EXPANDS a set of highly correlated inputs rather than compressing them. The symptom was already in the logs and had been read as a depth problem: the shallowest topology consistently beat the deepest (perceptron 52.7% balanced, hybrid 41.3%). Over-parameterization predicts that ordering just as well as covariate shift does, and only one of the two had been addressed. ComputeFirstLayerWidth() budgets roughly one first-layer weight per in-sample bar. Measured across the configurations in use: M15 10y -> 256 units, 129,071 weights, 0.73 per bar H1 10y -> 64 units, 28,727 weights, 0.65 per bar H4 10y -> 16 units, 7,559 weights, 0.68 per bar Two design points that matter: - It estimates in-sample bars from the STUDY PERIOD and timeframe, not from Bars(). What is downloaded grows over a terminal's lifetime, and a topology that widened as history filled in would re-key its own weights file and discard a trained model. - The result is snapped down to a coarse power-of-two ladder, so the estimate would have to be wrong by ~2x to change the answer. Every field it reads is already part of the weights-filename fingerprint, so the derived value needs no fingerprint entry of its own. The public setter is removed - it could only have been called after construction, and would either be ignored or silently re-key the model mid-run. Where the data cannot support even the floor (D1 over 10 years is under 2,000 bars) it now says so and names the fixes, rather than quietly training a model with more weights than examples. The DB config fingerprint drops the term too, which re-keys existing pattern databases once - correct, since a model an order of magnitude smaller should not inherit the old one's win-rate history. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 13:01:16 -04:00
//--- Width of the first dense layer, DERIVED rather than configured. It used to be an input
//--- (FIRST_LAYER_NEURONS, default 500) whose only sensible value depends entirely on two things the
//--- user cannot see: how wide the input vector ended up after feature selection, and how much
//--- in-sample data the study period actually yields. Left to a hand-picked constant it was badly
//--- wrong - 500 units against a 420-wide input is 210,500 weights, 72% of a 292,583-weight model,
//--- against ~36,500 training bars of which only ~2,236 are directional. That is ~8 parameters per
//--- sample, and it EXPANDS a set of highly correlated inputs instead of compressing them. The
//--- symptom is already in the logs: the shallowest topology consistently beat the deepest, which is
//--- what over-parameterization looks like from the outside.
//--- Computed from the STUDY PERIOD rather than from bars currently downloaded, so the answer is a
//--- deterministic function of the inputs and cannot drift as history fills in - and is then snapped
//--- to a coarse power-of-two ladder so even a large error in the estimate lands on the same rung.
//--- Every field it reads is already part of the weights-filename fingerprint, so the derived value
//--- needs no fingerprint entry of its own. MUST be called before the fingerprint is built and never
//--- again (see the note on fingerprint-feeding members at the top of this file).
int ComputeFirstLayerWidth(void) const;
feat(nn): derive conv filter count and LSTM hidden size from the data Same defect the first-layer width had before 2026-07-29: both were inputs whose defaults were fixed constants picked with no reference to the input they sit on, which is the only thing that decides whether either number is sane. The conv layer is a per-bar projection - AddConvStage sets window = step = one bar's features - so its filter count should be read against the per-bar feature count. Sixteen filters COMPRESSED a 50-feature configuration 3x but EXPANDED a minimal 4-feature one 4x, and the expanding case adds parameters below every learnable layer without adding information. Now derived as half the per-bar feature count, snapped down a power-of-two ladder. The LSTM stage was the bigger miss. Its weight count is exactly 4*H*(H+inputs+1) (CNeuronLSTMOCL::SetInputs) and AddLstmStage feeds it the whole flattened vector, so the shipped 32 units against a 540-wide input is ~73k weights - more than DOUBLE the entire derived dense taper it feeds. It was the one stage the capacity budget never covered, which is why deriving the dense stack alone did not stop LSTM and HYBRID from being over-parameterized. Now solved from the same one-weight-per-in-sample-bar budget the first layer spends. Factored EstimatedInSampleBars() out of ComputeFirstLayerWidth so all three decisions spend one budget rather than each guessing at the training-set size separately. Both new values are assigned alongside the first-layer width, before the fingerprint that hashes them, and are functions of inputs already in that hash - so they need no entry of their own, and the same reasoning removes them from the DB config key. Both builds compile 0 errors, 0 warnings. Re-keys existing models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 09:22:11 -04:00
//--- Expected in-sample training rows for the configured study period, split and timeframe. Factored
//--- out of ComputeFirstLayerWidth so every derived capacity decision spends the SAME budget - three
//--- stages each guessing at the training-set size independently is how they drift apart.
double EstimatedInSampleBars(void) const;
//--- Conv output-filter count and LSTM hidden width, DERIVED for the same reason the first-layer width
//--- is. Both were inputs whose defaults (16 filters, 32 units) were fixed constants picked without
//--- reference to the input width they sit on or the data available to fit them - so on a minimal
//--- feature set the conv stage EXPANDED the input, and the LSTM block quietly carried more weights
//--- than the entire dense taper below it. Both MUST be called before the fingerprint is built and
//--- never again: they assign fingerprint-feeding members (see the note at the top of this file).
int ComputeConvFilterCount(void) const;
int ComputeLstmHiddenSize(void) const;
feat(nn): derive dense depth, train on all history, pin the shape in .cfg Completes the derived-topology work. Three inputs removed. AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five entries instead of eight. Depth is now derived from the two endpoints the taper already has to connect (derived first-layer width, output-tied final width) at a 2x per-layer compression target, clamped [2..5]. Asking a user to pick a layer count while the code derives the widths those layers taper between was asking for half a decision: at 64 units tapering to 12, four layers compress by 1.4x per step and five by 1.3x, so the extra depth bought no abstraction. On the shipping H1/10y default the derivation lands on 3 layers - the depth that actually won Run 2. StudyPeriods removed. There is no case for training on less data than the broker provides at a ~6% directional base rate; the honest generalization read comes from the OOS holdout, not from withholding history. Training now starts at the earliest available bar, floored by MinTrainYear, which answers a different question (excluding dubious pre-history) and stays. That required closing the hazard the old code documented: the capacity budget now MEASURES the symbol's real bar count, and a topology derived from a measurement would widen as history downloads. Both ends are now pinned. Every derived value left the weights-filename fingerprint - keying a filename on a measured quantity means the EA looks for a file that does not exist, starts from era 0 and orphans a trained model, silently, because a missing cache is the normal first-run state. The shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the four derived fields rather than diffing them; a mismatch there would discard a fully-trained model over nothing the user did. Two fields appended to the .cfg for the conv/LSTM stages, length-guarded on read because FileReadInteger past EOF returns 0 with no error. ForceHiddenLayers, a compile-time constant like DebuggingMode, pins depth for diagnostic comparisons. It joins the fingerprint only when non-zero, so forced depths get their own files - sequential comparisons only, not simultaneous from one .ex5. Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64, 3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from ~58k to ~28k weights. Both builds compile 0 errors, 0 warnings. Re-keys existing models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
//--- Dense-taper DEPTH, derived 2026-07-30 from the two endpoints the taper connects. It was the depth
//--- suffix on each AI_CHOICE entry (MLP_3L/MLP_4L/..._2L); asking a user to pick a layer count while
//--- the code derives the widths those layers taper between is asking for half a decision. Reads
//--- m_initialNeuronsCount, so it MUST be called after ComputeFirstLayerWidth and before the
//--- fingerprint - see the note on fingerprint-feeding members at the top of this file.
int ComputeHiddenLayerCount(void) const;
fix: stop a .nnw from pinning a superseded architecture A .nnw persists the ARCHITECTURE, not just the weights: Save writes (int)activation per neuron and Load reads it straight back. The activation chosen in BuildFreshTopology() therefore only ever reached a brand-new topology - every reload restored the file's value and the next save wrote it back out, so a wrong value could never heal while the source read as though it were already fixed. That is how five models kept training with an unbounded NONE classification head for a full day after the 07-28 revert to SIGMOID. Confirmed by parsing the binaries: 848cb42c.nnw / 2e754b43.nnw carry `act=NONE` on the 3-neuron output layer, while a genuinely reset model of the same config carries act=SIGMOID. In the log it showed as negative "OOS raw out" values - impossible under sigmoid - escalating to a 4.14e13 logit spread with all three classes numerically identical (input-independent output) and balanced accuracy pinned on the 33.3% one-class floor. - OutputLayerActivation() is now the single source of truth, called by both BuildFreshTopology() and the new load-time repair, so the two can no longer diverge the way a duplicated literal did. - CNet::EnforceOutputActivation() re-asserts it after Load and reports the stale value; CExpertSignalAIBase::EnforceTopologyContract() logs the repair loudly, since weights learned under the old head may not be worth keeping even once the head is corrected. - Hidden layers are deliberately left alone: they legitimately differ per stage (PRELU dense/conv, NONE pool, TANH LSTM). Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 12:00:40 -04:00
//--- Re-assert everything about a just-loaded net that lives in the FILE but is owned by the CODE.
//--- Call after every successful Net.Load(); no-ops (and stays silent) when the file already agrees.
void EnforceTopologyContract(void);
//--- common network bootstrap: indicators, topology build/load, training-file bookkeeping
bool InitNeuralNetwork(CIndicators *indicators);
//--- Exclusive per-config claim, so two charts can never train into one set of model files. Every
//--- retrain-affecting input is already hashed into m_activeFileName, so "same file" IS "same
//--- config" - which makes the filename the only correct lock identity. Live charts only: each
//--- tester/optimizer agent is a separate process with its own sandboxed _optcache copy, and they
//--- are *meant* to run the same config in parallel.
bool AcquireConfigLock(void);
void ReleaseConfigLock(void);
void DrawObject(datetime time, double signal, double high, double low);
void DeleteObject(datetime time);
//--- Time-ordered NMS sweep over m_arrowSignalCache: prunes each same-direction run down to its
//--- earliest bar (deleting redundant neighbors within m_signalClusterWindow). Run once per era end.
void PruneDirectionalClusters(int bars);
feat: 10-bar decluster window + alternation on every signal consumer SignalClusterWindow 3 -> 10 for all topologies. On H1 a 3-bar window collapsed only the tightest runs and left visible clusters at every turn; 10 bars is closer to the spacing of genuinely distinct setups. ALTERNATION. Rule 1 only collapses a same-direction run INSIDE the window; past it a second Buy is emitted with no Sell between, giving Buy/Buy/Buy/Sell. With both directions tradeable that sequence is the model re-entering a move it is already in rather than finding a new one. The kept sequence must now alternate: the first signal passes, and after that a direction passes only if the last KEPT signal was the opposite one. Added to ALL THREE consumers, with identical logic, because they must agree: - NmsLiveAccept -> the live trade - pass 3's OOS replay -> the tally the deploy gate grades - PruneDirectionalClusters -> the drawn history A rule applied to only some of these certifies one strategy and trades another - the same defect class as the geometry the gate certified while OpenParams placed something else (9a7c37f) - and would draw the user arrows the EA would never have taken. Deliberately NOT applied to the LABEL. The barrier target has no "must flip" invariant: consecutive Buy labels are routinely correct, and an earlier alternation gate was removed with the triple-barrier relabel for exactly that reason. This filters what is ACTED ON, which is what "applies to training" can honestly mean here - pass 3's declustered tally is the training-side number that decides deployment. BothDirectionsTradeable() is the stated precondition (with one side disabled there is no opposite to wait for, so alternation would suppress everything after the first call). This build has no long-only/short-only input, so it is constant true - kept as a named predicate so a future direction restriction has one place to change rather than three call sites silently assuming both sides. Build tag -> nms-alternate-v4. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 14:26:12 -04:00
//--- Whether BOTH directions can currently be traded, which is the precondition for the alternation
//--- rule in the NMS paths: with only one side enabled there is no opposite signal to wait for, so
//--- requiring alternation would suppress everything after the first call. This build has no
//--- long-only/short-only input - the AI head emits both and nothing downstream restricts by side -
//--- so it is constant true. Kept as a named predicate rather than folded away so the precondition
//--- is stated where the rule reads it, and adding a direction restriction later has one place to
//--- change instead of three call sites that silently assume both sides.
bool BothDirectionsTradeable(void) const { return true; }
//--- Live newest-bar NMS accept test (time-keyed, idempotent per bar time - see m_signalClusterWindow).
bool NmsLiveAccept(datetime barTime, ENUM_SIGNAL dir, double conf)
{
if(m_signalClusterWindow <= 0)
return true;
if(dir != Buy && dir != Sell)
return true;
// Idempotent re-eval of the same bar (RefreshLatestSignal can run more than once per bar).
if(dir == Buy && m_nmsLiveBuyTime == barTime)
return m_nmsLiveBuyAccept;
if(dir == Sell && m_nmsLiveSellTime == barTime)
return m_nmsLiveSellAccept;
long minGap = (long)m_signalClusterWindow * PeriodSeconds();
datetime lastSame = (dir == Buy) ? m_nmsLiveBuyTime : m_nmsLiveSellTime;
bool accept;
// 1) Same-direction contiguous collapse: suppress if within the window of the previous SEEN
// same-direction bar (advance last-seen below either way, so a whole run collapses to one).
if(lastSame != 0 && (long)(barTime - lastSame) <= minGap)
accept = false;
else
{
// 2) Cross-direction resolution vs the last KEPT opposite signal: keep the stronger side.
accept = true;
if(m_nmsLiveKeptTime != 0 && m_nmsLiveKeptDir != dir &&
(long)(barTime - m_nmsLiveKeptTime) <= minGap)
{
if(conf > m_nmsLiveKeptConf)
DeleteObject(m_nmsLiveKeptTime); // this bar is stronger: remove the weaker opposite arrow
else
accept = false; // the kept opposite is stronger: suppress this bar
}
feat: 10-bar decluster window + alternation on every signal consumer SignalClusterWindow 3 -> 10 for all topologies. On H1 a 3-bar window collapsed only the tightest runs and left visible clusters at every turn; 10 bars is closer to the spacing of genuinely distinct setups. ALTERNATION. Rule 1 only collapses a same-direction run INSIDE the window; past it a second Buy is emitted with no Sell between, giving Buy/Buy/Buy/Sell. With both directions tradeable that sequence is the model re-entering a move it is already in rather than finding a new one. The kept sequence must now alternate: the first signal passes, and after that a direction passes only if the last KEPT signal was the opposite one. Added to ALL THREE consumers, with identical logic, because they must agree: - NmsLiveAccept -> the live trade - pass 3's OOS replay -> the tally the deploy gate grades - PruneDirectionalClusters -> the drawn history A rule applied to only some of these certifies one strategy and trades another - the same defect class as the geometry the gate certified while OpenParams placed something else (9a7c37f) - and would draw the user arrows the EA would never have taken. Deliberately NOT applied to the LABEL. The barrier target has no "must flip" invariant: consecutive Buy labels are routinely correct, and an earlier alternation gate was removed with the triple-barrier relabel for exactly that reason. This filters what is ACTED ON, which is what "applies to training" can honestly mean here - pass 3's declustered tally is the training-side number that decides deployment. BothDirectionsTradeable() is the stated precondition (with one side disabled there is no opposite to wait for, so alternation would suppress everything after the first call). This build has no long-only/short-only input, so it is constant true - kept as a named predicate so a future direction restriction has one place to change rather than three call sites silently assuming both sides. Build tag -> nms-alternate-v4. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 14:26:12 -04:00
// 3) ALTERNATION. Rule 1 only collapses a same-direction run inside the window; past it, a
// second Buy is emitted with no Sell in between, giving Buy/Buy/Buy/Sell. When BOTH directions
// are tradeable that sequence is the model re-entering a move it is already in, not finding a
// new one. Require the kept sequence to alternate: the first signal of a run passes (nothing
// to alternate with), and after that a direction only passes if the last KEPT signal was the
// opposite one.
// Gated on both directions being enabled - in a long-only or short-only configuration there
// is no opposite signal to wait for, so this would suppress everything after the first.
// NOTE this is deliberately a POST-PROCESSING rule, not a change to the label. The barrier
// target has no "must flip" invariant - consecutive Buy labels are routinely correct - and an
// earlier alternation gate was removed with the triple-barrier relabel for exactly that
// reason. What alternates is what gets ACTED ON: arrows, the pass-3 declustered tally the
// deploy gate grades, and the live trade.
if(accept && BothDirectionsTradeable() && m_nmsLiveKeptTime != 0 && m_nmsLiveKeptDir == dir)
accept = false;
}
if(dir == Buy)
{
m_nmsLiveBuyTime = barTime;
m_nmsLiveBuyAccept = accept;
}
else
{
m_nmsLiveSellTime = barTime;
m_nmsLiveSellAccept = accept;
}
if(accept)
{
m_nmsLiveKeptTime = barTime;
m_nmsLiveKeptDir = dir;
m_nmsLiveKeptConf = conf;
}
return accept;
}
fix(chart): arrows survived the EA that drew them - persist, then clear Reported: on deinit the panel and status label go, the signal arrows stay. Two independent causes, both fixed here. 1. It was partly deliberate. ShutdownChartCleanup carried a second behaviour selected by a `preserveChartArrows` flag derived from the deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the arrows were left on the chart on purpose, to avoid a reload flicker. That branch IS the reported symptom, an operator cannot tell it apart from a cleanup that failed, and it was outright wrong whenever the reload changed the config - REASON_PARAMETERS means exactly that, and the preserved arrows then belonged to a model the chart no longer runs, with nothing marking them stale. It is gone, along with the flag and m_purgeChartOnDestruct. One path now: persist, clear, restore on the next attach. 2. Whatever remains was unfalsifiable. PurgeChart was a single ObjectsDeleteAll(prefix) whose return value was discarded, with no caller ever looking at the chart again - so "the arrows are still there" and "the arrows were never there" produced identical evidence, which is why the report survived three sessions. It now verifies: after the bulk delete it walks the OBJ_ARROW-typed list (a handful of objects, not the whole chart), deletes any surviving WarSig_ by name, and says so. Costs one typed scan when the bulk delete works, which is the normal case; names the root cause when it does not. Every failure mode of SaveChartSignals was also silent - it returned void and had three bare early returns. It returns bool now, logs the open error with the filename, and the shutdown purge is CONDITIONAL on it: for a converged model the chart objects are the only copy of its signal history (nothing redraws them - the renderer runs per training era and a deployed model has none left), so a chart left littered because the disk write failed beats a clean chart bought by destroying the history. Either way the log now says which happened. Also states the user's rule once, where arrows come back rather than across InitNeuralNetwork's several exits: no weights loaded for this config => clear the sidecar and start visually clean. A fresh run must not inherit calls it never made, and the first save would otherwise adopt them (the sidecar is rebuilt by scanning the chart). Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
int PurgeChart(void);
ENUM_SIGNAL DoubleToSignal(double value);
//--- Shared status-label formatting for all three of Train()'s era passes (pass 1 sequential scan/
//--- display, pass 2 shuffled backProp, pass 3 post-training OOS scoring) - see m_isTrainQueue's and
//--- m_isPass2Active's declaration comments for why the era loop is now three passes instead of one.
//--- Extracted so the panel keeps updating every bar throughout ALL three passes instead of freezing
//--- during pass 2/3 the way it did when this was pass 1-only inline code - a stalled-looking panel
//--- during a still-running era was reported as looking "stuck". Throttled internally (see
//--- m_lastStatusLabelUpdateTick) - SetStatusLabel() (System\StatusLabel.mqh) does real text-layout
//--- work (TextGetSize/WrapLineInto per line) AND an unconditional ChartRedraw(0) every call, which
//--- gets slower as more chart objects accumulate over a long backtest. Calling it unconditionally
//--- once per bar was already the existing (pass-1-only) behavior; doing that again independently in
//--- pass 2 AND pass 3 roughly TRIPLED total ChartRedraw() calls per era and was the actual cause of
//--- a training run observed taking 1.5+ hours without completing era 0 - not the shuffle itself.
//--- forceRefresh=true bypasses the throttle below - used exactly once per era, right after the
//--- era-end block (Train()) finalizes m_eraCount/dOosForecast, so the panel's "Era %d"/"OOS Acc"
//--- fields update in the SAME moment the console's "training in progress - era N" line does.
//--- Without it, the panel's last real redraw during a normal (throttled) bar-scan call happened
//--- DURING pass 3, before m_eraCount++ - so it kept showing era N-1's number (with what was, by
//--- then, already era N's near-final accuracy) for this era's entire duration, only catching up to
//--- the correct era number once the NEXT era's own bar-scan calls started - a full one-era-behind
//--- display lag relative to the console log, reported in practice.
void UpdateTrainingStatusLabel(const string &progressLine, double neuron0, double neuron1, double neuron2, double signalValue, bool forceRefresh = false);
//--- Per-instance (NOT a function-local static - see CExpertSignalCustom::Direction()'s declaration
//--- comment for why that distinction matters for a method shared across PAI/CONV/LSTM instances)
//--- wall-clock throttle gate for UpdateTrainingStatusLabel()'s ChartRedraw().
uint m_lastStatusLabelUpdateTick;
//--- Last values passed to UpdateTrainingStatusLabel() - cached (updated on EVERY call, throttled
//--- or not) so the forced era-end refresh above has something real to redraw with instead of a
//--- stale/zeroed placeholder, since no "current bar" exists once an era's own three passes are done.
double m_lastDisplayNeuron0, m_lastDisplayNeuron1, m_lastDisplayNeuron2, m_lastDisplaySignal;
//--- Latest OOS Buy/Sell recall (-1 = n/a, same convention as logBuyRecallPct etc.), cached the same
//--- way as m_lastDisplayNeuron0 above so UpdateTrainingStatusLabel() can show it on every call, not
//--- just the era-end one that actually just computed it. Surfaced on-chart (not just the Experts
//--- log) because a Buy/Sell-diluted-by-Neutral headline accuracy number is what a trader watching
//--- the panel sees by default, but Buy/Sell recall is what actually predicts trading performance -
//--- Neutral is "don't trade," so a model can look good on blended accuracy purely by calling Neutral
//--- often, while its actual Buy/Sell calls are unreliable.
int m_lastBuyRecallPct, m_lastSellRecallPct;
//--- turns the classification output layer's 3 values (TempData[0..2], SIGMOID activation - see
//--- BuildFreshTopology() - each already in [0,1]) into a softmax probability distribution in
//--- place, and returns the signed dPrevSignal convention (+P(buy), -P(sell), 0.0 exactly for
//--- neutral) used by both Train()'s live-forecast branch and RefreshLatestSignal(). Softmax is
//--- monotonic per-element so it can't change which of the 3 wins - it only exists to turn the 3
//--- independently-trained values into a normalized confidence. Max-subtracted before exp() for
//--- numerical stability regardless (safe since softmax(x) == softmax(x - max(x))).
double ApplyClassificationSoftmax(void);
//--- Post-hoc logit adjustment / prior correction: reads the raw softmax probabilities
//--- ApplyClassificationSoftmax() just left in TempData[0..2] and returns the PRIOR-CORRECTED signed
//--- decision (same +P'(buy)/-P'(sell)/0-neutral convention). This is the exact rule live trading
//--- fires on and the rule the live-fired precision metric scores. See the definition for the math.
double AdjustedSignalFromSoftmax(void);
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
//--- Margin between the winning class and its best rival, from the softmax already in TempData.
//--- Returns <0 when the winner is Neutral (not a directional call, so no operating point applies)
//--- or when the outputs are unreadable. This is the statistic the threshold is expressed in - NOT
//--- the winning probability on its own, which moves with how confident the net is overall rather
//--- than with how CLOSE the decision was, and would therefore drift as calibration changes.
double DirectionalMargin(void);
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- Reset / accumulate / fit, in the order the calibration walk calls them. See
//--- DIR_CONF_THRESHOLD_BINS and DIR_CONF_CALIB_PCT_OF_IS.
feat: excursion-size head (Stage 1, measurement only) Direction is closed - normalised asymmetry fails on three instruments with a working positive control, and the classifier's own best-of-999 era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is a different question and RANGE clears at ~4x its null. Checked the denomination before building on that, since the source memo warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is predictable" is a claim about travel RELATIVE to current ATR, not a restatement of "ATR is autocorrelated". It is exactly the part a fixed multiple (stop 3.31*ATR, target 1.64*ATR) discards. A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches ladder rung k) upward and downward. Survival parameterisation rather than regressing the multiple, because it needs nothing new from CNet: sigmoid outputs and the per-neuron delta the `total != 3` branch already applies (a quantile head would need a linear activation and a pinball gradient in Network.mqh, Network.cl and the DirectML path, on a class four topologies share). Targets are free - m_ladderUpAt already records first-touch age per rung with 0 meaning never reached. Separate net, not extra outputs on the classifier: more outputs would change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push the count off 3 - the exact condition backProp uses to select the joint softmax gradient the 3-class head depends on. The classifier is bit-for-bit unaffected and this is removable without trace. STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the constant per-rung base rate - the baseline a fixed ATR multiple already assumes - with both predictors fitted IS and evaluated OOS, so neither gets a look at the test set. Positive skill justifies Stage 2 (drive SL/TP and sizing off ExcursionQuantile, which is defined and deliberately uncalled). Zero or negative means ATR already carries everything and Stage 2 must not be built. Trains only on primary occurrences: the replay queue oversamples for CLASS balance, and a direction-balanced sample is a biased SIZE sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
//--- EXCURSION-SIZE HEAD - see Expert\AIBase\Excursion.mqh. Predicts how FAR price travels, never
//--- which way; Stage 1 measures whether it beats a constant ATR multiple and places no orders.
bool ExcursionBuildTopology(CArrayObj &topology);
bool ExcursionEnsureHead(void);
bool ExcursionTargets(int idx);
void ExcursionTrainStep(int idx);
void ExcursionScoreStep(int idx);
feat: race the excursion head against a trailing-quantile incumbent Beating a frozen global constant is the weakest admissible bar for replacing a global constant. The honest incumbent is a rolling rung frequency: it adapts to the volatility regime - exactly what the head claims to predict - and needs no model, no 760 inputs and no training. Implemented as a ring of per-bar outcome bitmasks (32 rungs fit one ulong), sized horizon + EXCURSION_TRAIL_WINDOW. The newest `horizon` entries are held back UNRESOLVED: a bar's rung outcomes are only known one horizon later, so using them would be lookahead and would flatter the incumbent into an opponent the head could never fairly beat. Pass 3 walks oldest-to-newest, so "pushed more than horizon bars ago" is exactly "resolved by now". Each push is O(rungs), not O(window). The head's decision-rung Brier is pro-rated to the trailing estimate's coverage before the ratio, since the incumbent only scores bars where its window is warm. This line is worth reading on its own, independently of the head: if the trailing quantile beats the global constant, that is a cheap risk-control win available with no machine learning at all - and it is the same number either way, so the run answers both questions in one pass. The ring is deliberately NOT reset per era - it estimates the market, not the era, and re-warming 500 bars every era would leave the incumbent unusable over the first chunk of every scoring pass, handing the head a free win on exactly those bars. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 15:57:11 -04:00
void ExcursionTrailPush(void);
feat: excursion-size head (Stage 1, measurement only) Direction is closed - normalised asymmetry fails on three instruments with a working positive control, and the classifier's own best-of-999 era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is a different question and RANGE clears at ~4x its null. Checked the denomination before building on that, since the source memo warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is predictable" is a claim about travel RELATIVE to current ATR, not a restatement of "ATR is autocorrelated". It is exactly the part a fixed multiple (stop 3.31*ATR, target 1.64*ATR) discards. A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches ladder rung k) upward and downward. Survival parameterisation rather than regressing the multiple, because it needs nothing new from CNet: sigmoid outputs and the per-neuron delta the `total != 3` branch already applies (a quantile head would need a linear activation and a pinball gradient in Network.mqh, Network.cl and the DirectML path, on a class four topologies share). Targets are free - m_ladderUpAt already records first-touch age per rung with 0 meaning never reached. Separate net, not extra outputs on the classifier: more outputs would change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push the count off 3 - the exact condition backProp uses to select the joint softmax gradient the 3-class head depends on. The classifier is bit-for-bit unaffected and this is removable without trace. STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the constant per-rung base rate - the baseline a fixed ATR multiple already assumes - with both predictors fitted IS and evaluated OOS, so neither gets a look at the test set. Positive skill justifies Stage 2 (drive SL/TP and sizing off ExcursionQuantile, which is defined and deliberately uncalled). Zero or negative means ATR already carries everything and Stage 2 must not be built. Trains only on primary occurrences: the replay queue oversamples for CLASS balance, and a direction-balanced sample is a biased SIZE sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
void ExcursionResetEraScores(void);
double ExcursionQuantile(bool upward, double tau);
void ExcursionReport(void);
feat: fitted directional confidence threshold - selectivity gets a mechanism The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 15:04:37 -04:00
void ResetDirConfHistogram(void);
void AccumulateDirConfSample(double margin, bool wasCorrect, bool isPrimaryBar);
void FitDirConfThreshold(void);
fix: the operating point was fitted on bars the net had memorized FitDirConfThreshold harvested its margin histogram from pass 2's own backprop samples. Pairing every fit against the same era's OOS result shows what that measured: PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp The gap grows monotonically while OOS stays flat, so within a handful of eras the curve stops describing behaviour on unseen bars. That is fatal here specifically, because the objective branches on the SIGN of (p - break-even): the memorized curve reads +12pp at 95% coverage, so coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire on every bar. The "p < p0 -> get more selective" branch, which is the actual regime and the entire point of 983a6a3, could never fire because IS never showed p < p0. Carve a calibration slice out of the IS span - DIR_CONF_CALIB_PCT_OF_IS, purged from backprop by one label horizon on BOTH sides (the far-side purge is not optional: without it the newest training bars carry labels partly decided by price action inside the slice, putting the memorization straight back into the curve). Score it in a new chunked pass 2.5, after pass 2 has trained and before pass 3 grades - the only position where the histogram is simultaneously not-trained-on, not-graded, and current with the weights it will be applied to. Costs 15% of the training data. Worth it beyond honesty: the deploy gate needs dirPrecPct > chance + EDGE_MIN_SIGMAS*SE, and a threshold pinned near zero dilutes any edge concentrated in the confident bars across every bar the model calls, driving dirPrecPct toward chance by construction. A threshold that can be selective is the only mechanism by which a small, concentrated edge could ever clear that gate. Also: a sparse histogram now KEEPS the previous threshold instead of resetting to 0.0. A failed measurement must not decay to the most exposed setting in the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 15:58:18 -04:00
//--- CALIBRATION BAND BOUNDS, in pass-1 bar indices (0 = newest bar, so LARGER index = OLDER).
//--- The era's bars lay out, newest to oldest:
//---
//--- [0, oosCutoff) OOS - graded by pass 3, never trained on
//--- [oosCutoff, calibLo) purge - one label horizon, discarded entirely
//--- [calibLo, calibHi) CALIBRATION - fits the threshold, never trained on
//--- [calibHi, calibHi + horizon) purge - one label horizon, discarded entirely
//--- [calibHi + horizon, ...) IS - the backprop queue
//---
//--- A purge on BOTH sides, not just the OOS one: a triple-barrier label is decided by the horizon
//--- bars that FOLLOW its bar, so without the far-side purge the newest training bars would carry
//--- labels partly determined by price action inside the calibration slice - the same Lopez de Prado
//--- ch. 7 leak the OOS boundary already guards against, and it would put the memorization straight
//--- back into the curve this slice exists to keep clean.
int CalibPurgeBars(void) const { return (int)MathMax(m_barrierHorizonBars, 1); }
int CalibLoIndex(int oosCutoff) const { return oosCutoff + CalibPurgeBars(); }
//--- Zero (an empty band) whenever the era is too short to carve one without eating the training set;
//--- callers must treat that as "no calibration this era" and leave the threshold where it is.
int CalibBandBars(int totalIter, int oosCutoff) const
{
int isSpan = totalIter - CalibLoIndex(oosCutoff) - CalibPurgeBars();
if(isSpan <= 0)
return 0;
return (int)(isSpan * (DIR_CONF_CALIB_PCT_OF_IS / 100.0));
}
int CalibHiIndex(int totalIter, int oosCutoff) const
{ return CalibLoIndex(oosCutoff) + CalibBandBars(totalIter, oosCutoff); }
//--- EMA-updates the persisted true class base rates (m_priorBuy/Sell/Neutral) from a just-finished
//--- era's true class counts. No-op on an empty/degenerate tally.
void UpdateClassPriors(long buyCnt, long sellCnt, long neutralCnt);
feat(ai): logit-adjusted loss, replacing oversampling and the post-hoc prior Menon et al. 2021 (ICLR), "Long-tail learning via logit adjustment": add tau*log(prior_c) to each class logit inside the training gradient. Softmax CE on adjusted logits is consistent for BALANCED error - the metric checkpoint selection already ranks on - so the loss and the deploy decision finally optimize the same thing. The engine already computed a true softmax + categorical-CE gradient and wrote it over the per-neuron sigmoid delta, so this is an offset added to three logits in the two places that gradient is built (backProp scalar path and backPropOCL). No backend, kernel or DLL change; the forward pass and every inference path are untouched, which is the point - the network learns to absorb the offset, so its raw argmax becomes the balanced-optimal decision with nothing applied at inference. Replaces rather than stacks. Minority replay is disabled while this is on, and the post-hoc inference prior is forced off. Stacking is not a theoretical worry: simulated on the measured 1118/1119/34298 distribution in the weak-signal regime, plain CE collapses to Neutral (33.4% balanced, Buy 0%), replay reaches 48.1%, logit adjustment 50.9% with better balance - and BOTH together score 45.4% with Neutral recall at 0%, worse than either alone. Buda et al. 2018 predicts exactly that. Motivation from the six-chart run: every topology took one direction to ~50% recall and abandoned the other, the direction chosen arbitrarily (the batch-norm control went Buy 1% / Sell 42%, the inverse of the other five). One era in 1,301 cleared the per-class recall floor. Fingerprinted conditionally, so the converged 60.7% models on disk keep their filenames and stay loadable as the fallback. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:05:14 -04:00
//--- Installs tau*log(prior_c) on Net from the freshly measured priors. Called once per era
//--- start, straight after UpdateClassPriors, so the offsets track the same distribution the
//--- era is scored against. No-op (and actively clears stale offsets) when the input is off.
void ApplyLogitAdjustment(void);
//--- Small binary sidecar (fileName + ".stats") persisting the calibration state that must survive a
//--- restart for live trading to behave like training: the true class priors and m_confidenceCalScale.
bool SaveModelStats(string fileName, bool common);
bool LoadModelStats(string fileName, bool common);
//--- Deploy-time (chart, backend present) self-check: runs the just-saved deployed model through both
//--- the backend and a temporary pure-MQL5 (CNet::SetCpuInference) clone on the same input window and
//--- returns true only if the outputs match within CPU_INFERENCE_MAX_DIFF. Gates whether an
//--- inference-only backtest may run DLL-free. Fails safe (returns false) on any error/mismatch or an
//--- architecture whose CPU path isn't ported yet - the caller then keeps the model on the DLL path.
bool ValidateCpuInference(void);
//--- Build the panel's "Buy/Sell accuracy: IS x% OOS y%" line (directional win-rate, Neutral excluded)
//--- from the cumulative counts (m_cumIsCorrect etc.); returns "...: measuring..." until at least one
//--- directional call has been validated. Shared by the training and live/complete simple panels.
string ComputeCompoundedAccuracyLine(void);
//--- Persist/restore the drawn directional arrows (the "WarSig_" objects) to a sidecar file so they
//--- survive an EA remove/re-add, recompile, or restart WITHOUT a retrain - the chart objects are
//--- destroyed on unload (destructor PurgeChart) and OnInit has no other way to bring them back.
//--- Stores each arrow's time/code/price and its hide state (OBJPROP_TIMEFRAMES), so the show/hide
//--- toggle is preserved too. Chart-only (a backtest has no persistent chart to restore to).
fix(chart): arrows survived the EA that drew them - persist, then clear Reported: on deinit the panel and status label go, the signal arrows stay. Two independent causes, both fixed here. 1. It was partly deliberate. ShutdownChartCleanup carried a second behaviour selected by a `preserveChartArrows` flag derived from the deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the arrows were left on the chart on purpose, to avoid a reload flicker. That branch IS the reported symptom, an operator cannot tell it apart from a cleanup that failed, and it was outright wrong whenever the reload changed the config - REASON_PARAMETERS means exactly that, and the preserved arrows then belonged to a model the chart no longer runs, with nothing marking them stale. It is gone, along with the flag and m_purgeChartOnDestruct. One path now: persist, clear, restore on the next attach. 2. Whatever remains was unfalsifiable. PurgeChart was a single ObjectsDeleteAll(prefix) whose return value was discarded, with no caller ever looking at the chart again - so "the arrows are still there" and "the arrows were never there" produced identical evidence, which is why the report survived three sessions. It now verifies: after the bulk delete it walks the OBJ_ARROW-typed list (a handful of objects, not the whole chart), deletes any surviving WarSig_ by name, and says so. Costs one typed scan when the bulk delete works, which is the normal case; names the root cause when it does not. Every failure mode of SaveChartSignals was also silent - it returned void and had three bare early returns. It returns bool now, logs the open error with the filename, and the shutdown purge is CONDITIONAL on it: for a converged model the chart objects are the only copy of its signal history (nothing redraws them - the renderer runs per training era and a deployed model has none left), so a chart left littered because the disk write failed beats a clean chart bought by destroying the history. Either way the log now says which happened. Also states the user's rule once, where arrows come back rather than across InitNeuralNetwork's several exits: no weights loaded for this config => clear the sidecar and start visually clean. A fresh run must not inherit calls it never made, and the first save would otherwise adopt them (the sidecar is rebuilt by scanning the chart). Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
bool SaveChartSignals(bool pruneChartObjects = true);
void LoadChartSignals(void);
fix(chart): arrows survived the EA that drew them - persist, then clear Reported: on deinit the panel and status label go, the signal arrows stay. Two independent causes, both fixed here. 1. It was partly deliberate. ShutdownChartCleanup carried a second behaviour selected by a `preserveChartArrows` flag derived from the deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the arrows were left on the chart on purpose, to avoid a reload flicker. That branch IS the reported symptom, an operator cannot tell it apart from a cleanup that failed, and it was outright wrong whenever the reload changed the config - REASON_PARAMETERS means exactly that, and the preserved arrows then belonged to a model the chart no longer runs, with nothing marking them stale. It is gone, along with the flag and m_purgeChartOnDestruct. One path now: persist, clear, restore on the next attach. 2. Whatever remains was unfalsifiable. PurgeChart was a single ObjectsDeleteAll(prefix) whose return value was discarded, with no caller ever looking at the chart again - so "the arrows are still there" and "the arrows were never there" produced identical evidence, which is why the report survived three sessions. It now verifies: after the bulk delete it walks the OBJ_ARROW-typed list (a handful of objects, not the whole chart), deletes any surviving WarSig_ by name, and says so. Costs one typed scan when the bulk delete works, which is the normal case; names the root cause when it does not. Every failure mode of SaveChartSignals was also silent - it returned void and had three bare early returns. It returns bool now, logs the open error with the filename, and the shutdown purge is CONDITIONAL on it: for a converged model the chart objects are the only copy of its signal history (nothing redraws them - the renderer runs per training era and a deployed model has none left), so a chart left littered because the disk write failed beats a clean chart bought by destroying the history. Either way the log now says which happened. Also states the user's rule once, where arrows come back rather than across InitNeuralNetwork's several exits: no weights loaded for this config => clear the sidecar and start visually clean. A fresh run must not inherit calls it never made, and the first save would otherwise adopt them (the sidecar is rebuilt by scanning the chart). Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
//--- The shutdown half of that pair: persist, THEN clear the chart, and report both counts. See the
//--- definition for why the order is fixed and why the clear is conditional on the write.
void PersistAndClearChartSignals(void);
//--- How many arrows the last successful SaveChartSignals() wrote - reporting only.
int m_lastArrowsSaved;
fix: the Adam second moment was never Adam - all four tiers Root cause of the B=32 regression, and it predates F4 entirely. Every Adam kernel stored v already square-rooted and then fed that stored value back in as if it were the variance: v_new = sqrt(b2 * v_old + (1 - b2) * g^2) That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below unit scale, so the denominator stops tracking the gradient and Adam degrades into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll (batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where a scale-invariant optimizer gives the same distance for both. After the fix all six magnitudes read 1.199 and v tracks |g| exactly. It hit conv/LSTM specifically because they sit behind a batch-norm with running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep in the degraded regime - while the dense stack near the loss stayed in the working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/ 0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already squared v back for gamma/beta and its comment named the kernels as wrong, which is exactly why gamma/beta kept training while the stages behind froze. Persisted .nnw needs no migration - v keeps its std-dev meaning. Also, the two ways F4 exposed it, both mine: - No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods (Krizhevsky 2014; Granziol et al. 2022), applied once in InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD. - Plateau patience denominated in eras, so raising B made the ladder 32x more impatient in its only unit. PAI converged at era 41 on ~49k updates where the same config had been finding new bests at era 1028. TrainPlateauPatienceEras() stretches it by the same sqrt(B). TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23 eras per stage, not 8 -> 45). Both helpers are identities at B=1. Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into the ranking key rather than checked at deploy time so a one-sided era cannot become best-so-far in the first place. Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded CAppDialog teardown that sat ahead of it - the same ordering inversion the rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit (vs ~1.1 s for the three that finished) having reached none of its cleanup, so its arrows stayed on the chart. Steps are now timed in the log. PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now walks every object type and reports the object counts when both are zero. Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt. FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 14:02:35 -04:00
//--- One-shot latch for PurgeChart()'s "saved N but the chart holds none" warning. PurgeChart runs
//--- twice on a clean removal - once from the shutdown path and again from the destructor, which is
//--- deliberate (the destructor covers teardowns that never reach OnDeinit) - and the second call
//--- necessarily finds an already-emptied chart with m_lastArrowsSaved still set. Without this latch
//--- that harmless second pass would report the discrepancy every single time and train the reader
//--- to ignore the one case where it is real.
bool m_purgeMismatchWarned;
fix: clear stale signal arrows when a fresh model starts at era 0 Arrow cleanup existed on two paths - the panel's reset-weights, and the topology-mismatch discard - but both are gated on there being a saved .nnw to delete. The third case had no cleanup at all: a fresh topology at era 0 with no weights behind it, which is what a changed config produces. A new fingerprint makes a new m_fileName, so the previous model's files are not "discarded", they are simply not this model's files, and nothing ever cleared the chart. That is not cosmetic. Arrows outlive the model that drew them twice over: 1. The chart objects live in the CHART, not the sidecar, so they survive a remove/re-add, a recompile, a restart and a fresh deploy no matter what happens to any file on disk. 2. SaveChartSignals() rebuilds the sidecar by SCANNING the chart for SIG_ARROW_PREFIX objects. So the first save of the fresh run adopts the dead model's calls and writes them out under the NEW model's filename - laundering them into the new model's history where nothing can separate them afterwards. Extracted the duplicated cleanup into ClearPersistedChartSignals(reason) - it cancels the deferred restore queue, deletes m_fileName + ".arrows", clears the namespaced chart objects and logs why - and called it from all three paths. The call sits at the BuildFreshTopology() call site, not inside it: the genetic tuner rebuilds a throwaway topology per candidate (AutoTune.mqh) and must never touch the chart. All three sites run after m_fileName has its config fingerprint appended, so they target the right sidecar. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 10:13:01 -04:00
//--- Wipe this model's drawn arrows AND their .arrows sidecar, plus any deferred restore still in
//--- flight. Call from every path that discards or replaces the trained weights - see the definition
//--- for why leaving them behind resurrects a dead model's calls through SaveChartSignals.
void ClearPersistedChartSignals(const string reason);
//--- Deferred ("async") half of LoadChartSignals: LoadChartSignals only PARSES the sidecar into the
//--- m_arrowRestore* buffers (an ~80KB read - instant) and returns, so OnInit never blocks; this then
//--- creates the chart objects in ARROW_RESTORE_BUDGET_MS slices, driven by the same 500ms timer that
//--- already paces training. MQL5 has no threads - a chart runs one thread - so blocking OnInit is what
//--- made the terminal look frozen (no panel, no status label, no journal) while ~2900 arrows were
//--- rebuilt. Time-boxed slices give the terminal room to paint the UI between them instead.
void AdvanceChartSignalRestore(void);
//--- parsed-but-not-yet-drawn arrows, consumed by AdvanceChartSignalRestore (see above)
datetime m_arrowRestoreTime[];
int m_arrowRestoreCode[];
double m_arrowRestorePrice[];
long m_arrowRestoreTf[];
int m_arrowRestoreIndex;
bool m_arrowRestorePending;
uint m_arrowRestoreStartMs;
//--- Deferred ("async") half of StartChartSignalRescan (public, defined inline further down): drains
//--- the per-bar inference loop in ARROW_RESTORE_BUDGET_MS slices off PollTraining's timer instead of
//--- blocking the button-click handler for however long a full lookback scan takes. Internal-only -
//--- called from PollTraining(), never from outside the class - so this stays protected while
//--- StartChartSignalRescan()/RescanPending() (which the panel button needs) are public.
void AdvanceChartSignalRescan(void);
int m_rescanIndex;
int m_rescanHi;
int m_rescanBarsNow;
bool m_rescanPending;
uint m_rescanStartMs;
//--- Raw (PRE prior-correction) argmax tally, accumulated per-bar across AdvanceChartSignalRescan's
//--- slices - lets the completion log distinguish "the network itself calls Neutral almost everywhere"
//--- from "the network still discriminates, but AdjustedSignalFromSoftmax's logit-prior correction is
//--- suppressing it down to Neutral" - both produce an identical all-Neutral m_arrowSignalCache/empty
//--- chart otherwise.
int m_rescanRawBuy;
int m_rescanRawSell;
int m_rescanRawNeutral;
bool ResizeBuffers(int barIndex);
bool RefreshData();
research: export the feature matrix and a raw OHLCV grid for offline work The bottleneck on this project has never been the modelling - it is that every hypothesis costs a compile, a deploy, an attach and a log read, and answers exactly one question. Days have gone into questions that are seconds of arithmetic once the data is in hand. Adds a RESEARCH-ONLY build, gated behind WARRIOR_EXPORT_FEATURES and never compiled into a shipped binary, which writes two things to Common\Files\Warrior_EA\Research\ and then does nothing at all: <symbol>_<tf>_features.csv - one row per bar: index, time, OHLC, ATR, and the m_neuronsCount feature values. Exactly what the network sees. The raw bars ride along on purpose: with OHLC and ATR offline, every barrier geometry, horizon and in-trade target is recomputable without MetaTrader in the loop. <symbol>_<tf>_rates.csv - raw OHLCV across a grid of 8 symbols x 5 timeframes. The 26 engineered features only exist for the attached chart (indicator handles bind to PERIOD_CURRENT); raw rates do not, so ONE attach yields the whole research grid. The bar time also makes session/hour/day-of-week derivable - the only inputs in play that are not a transform of the same OHLCV series. Safety, because this binary gets attached to a chart on a LIVE ACCOUNT to reach real history: - OnTick returns immediately, so Expert.OnTick() - the entire trading path - is unreachable regardless of the AlgoTrading toggle, the signal state or the inputs. Structurally incapable of sending an order, not merely unlikely to. - No config lock. It never trains and never saves a model, so it has nothing to protect against a concurrent chart - and taking the lock would make it refuse to start exactly when the config it wants to read is already open, which is when it is most useful. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 15:49:57 -04:00
#ifdef WARRIOR_EXPORT_FEATURES
//--- RESEARCH BUILD ONLY, never compiled into a shipped binary. Dumps exactly what the network sees -
//--- one row per bar: index, time, OHLC, ATR, then the m_neuronsCount feature values - to a CSV under
//--- Common\Files\Warrior_EA\Research\. Exporting the RAW BARS alongside the features is the point:
//--- with OHLC+ATR in hand every barrier geometry, horizon and in-trade target can be recomputed
//--- offline, so a research question costs seconds in Python instead of a compile/attach/read cycle.
void ExportFeatureMatrix(void);
//--- Raw OHLCV for a grid of symbols/timeframes - see the definition for why the grid is worth more
//--- than the engineered features on their own.
void ExportRawRates(void);
#endif
bool BufferTempData(int idx);
fix: the sequence models were reading the window backwards BuildFeatureWindow() replaces eight hand-rolled copies of the same loop and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first, because MQL5 timeseries indices run backwards and `r + b` with b ascending walks into the past. Harmless for PAI and CONV - a dense layer learns a weight per position either way, a conv learns time-mirrored kernels. Not harmless for the recurrent stacks: - LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t. - It writes output[] only when t == steps-1: the visible output IS the last hidden state. - c_t = f*c_{t-1} + i*g decays toward the start of the sequence. lstm_seq_flowcheck.cpp measured block 0's influence on the output at 1.2e-2 of block T-1's, at the shipped forget bias of 1.0. So the bar being PREDICTED sat at the far end of the decay and the output was handed to the OLDEST bar in the window - the exact inverse of what the window is for. ~80x backwards on LSTM and HYBRID, on all three tiers (OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never surfaced as a backend discrepancy. This does not create edge - the MI diagnostics read at the noise floor (p=0.4975) with a working positive control. It makes the one hypothesis those diagnostics explicitly do NOT cover testable: they are marginal and per-bar, and state they "cannot rule out one that only exists in combination or across time". The sequence model is the instrument for across-time structure and it has been crippled, so that hypothesis has never been honestly tested. Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and its features, so a stale .nnw would load cleanly and run a model fitted to one ordering against the other, silently. Re-keying every config is the point, not collateral damage. FORCES A FULL RETRAIN. Also: the now-relative bar caches are re-keyed on the two live paths. EnsureBarCachesCapacity() was only ever called from training paths, but once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar to RefreshConvergedSignal() and Train() is never re-entered - so nothing cleared the feature cache again for the life of the process. A chart that trained to convergence kept replaying the rows computed for the last training era's bar grid: the live signal froze at its convergence-time value, and OnlineLearnStep() backpropped those stale features against freshly resolved labels. Backtests were never affected (an inference-only process never allocates the arrays, so every read recomputes). Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-07 18:28:44 -04:00
//--- Assembles the full m_historyBars-wide input window ending AT bar r into TempData, OLDEST BAR
//--- FIRST. Use this everywhere instead of hand-rolling the loop: the chronological order is load-
//--- bearing for the LSTM/HYBRID stacks and cannot be enforced by convention across eight call
//--- sites. See the definition comment in AIBase\Features.mqh for the measurement behind that.
bool BuildFeatureWindow(int r);
//--- shared by OnTickHandler() and the timer-driven PollTraining() - see definition
void ScheduleTrainingIfNeeded(void);
void Train(datetime StartTrainBar = 0);
fix: a restart no longer loses the measured geometry or the training window Terminal restart, 22:25: all four resumed models sat on empty windows with enum 2:6 barriers. Three interlocking causes, all visible in one log excerpt: 1) THE PRE-SCAN WINDOW WAS SIZED BY THE SAVED WATERMARK. A resumed model's dtStudied sits at its last studied bar, so Bars(dtStudied, now) ~ 0 and the resumed-model MI pre-scan built a zero-bar "complete" label cache - logged as "Buy: 0 | Sell: 0 | Neutral: 0". Train()'s own era start RESETS dtStudied to the training-window rule before computing its window; the pre-scan did not. The rule is now factored into TrainWindowStart() and both use it. The scan also refuses to arm before SERIES_SYNCHRONIZED (it ran in the same second as OnInit), and deployed models keep their watermark - for them it gates inference recency, not a training window. 2) THE HORIZON LATCHED ON AN INDICATOR WARM-UP. ComputeBarrierHorizonBars ran against a ZigZag with 0 calculated legs, fell back, and EnsureBarrierHorizon latched fallback(32) x slMult x tpMult = 384 for the process lifetime. A leg-starved horizon is now PROVISIONAL: re-resolved on the next rebuild, the label cache wiped if it moved (labels from two horizons answer different questions), and the geometry deriver refuses to run from it - a pair derived over a warm-up window would get PINNED. 3) THE DERIVED GEOMETRY WAS NEVER PERSISTED. The .cfg is written at model creation and at weights-reset - both BEFORE era 0 derives - so the measured pair lived only in memory: every restart read back zeros, adopted nothing, fell back to the enum barriers, and the era-0-only gate meant a resumed model could NEVER re-derive. A full day of training on 3.33/1.62 resumed as 2:6. Now: the settled pair is pinned to the .cfg the moment derivation completes (one-shot, atomic write), and the derive gate accepts any model with no pinned pair, not just era 0 - mid-run stability is carried by m_geometryDerived itself, which never allows a second derivation. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 22:40:43 -04:00
//--- the training window's start time - shared by Train()'s era start and the label-cache pre-scan
datetime TrainWindowStart(datetime startTrainBar);
//--- outer loop around Train(): when AutoTuneIndicators is on, tries randomized AD indicator
//--- input variations across m_indicatorTuneTrials calls to Train(), keeping the best-OOS one
void TuneIndicatorsAndTrain(datetime StartTrainBar = 0);
fix: live inference queried the 1-tick forming bar - a window training never built RefreshLatestSignal ran at the first tick after a bar opens and built its window at r=0: series index 0 at that instant is a candle with one tick of data - (close-open)/atr ~ 0, high ~ low, degenerate volume, indicators on a 1-tick bar. Training never produces such a window (every labeled bar is fully closed, entry at that bar's CLOSE), so the deployed model's final timestep - the one the LSTM/HYBRID output is keyed to - was out-of-distribution on every live decision, and pass 3's deploy-gate OOS scores measured a different query than live executed. The parity index is r=1: the newest CLOSED bar, whose close IS the current price - the exact instant the label's hypothetical entry happens. Single backtests shared the old skew (same r=0), which is why the tester agreed with live while both disagreed with training. Bookkeeping split that the index change forces: m_lastBarTime/dtStudied stay anchored to the FORMING bar's open (they gate against SERIES_LASTBAR_DATE; anchoring at bar 1 would re-fire the refresh every tick), while bt - the arrow, its High/Low placement, and NMS declustering - anchors to the decision bar, now matching the rescan path's convention. Also: a failed refresh no longer trades the previous bar's signal for the whole bar. RefreshLatestSignal returns success, zeroes dPrevSignal on failure (no opinion beats a stale one), and RefreshConvergedSignal advances dtStudied only on success so the next tick retries - the tester path (m_lastBarTime) already worked this way; this is the live path catching up. FORCES RE-VALIDATION of deployed models: the effective live query distribution changes. Bundled with the backprop transpose fix's retrain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 18:10:23 -04:00
//--- recomputes dPrevSignal/chart arrow for the newest CLOSED bar (bar 1 - see the definition's
//--- 2026-08-11 parity comment); used after restoring a checkpointed model at the end of Train()
//--- so the live signal matches the deployed weights. Returns false when the window failed to
//--- build (dPrevSignal is zeroed, caller should not advance its new-bar watermark).
bool RefreshLatestSignal();
//--- inference-only "new bar" handler used once m_trainingComplete is true - see
//--- ScheduleTrainingIfNeeded()'s declaration comment for why this must NOT call Net.backProp()
void RefreshConvergedSignal(void);
//--- Online continual-learning step (live chart only) - see its implementation comment and the
//--- ONLINE_LEARN_* tunables. Backprops the deployed Net on bars whose ZigZag label has just become
//--- CONFIRMED (m_swingConfirmationBars matured), then blends the shadow under a rolling-accuracy
//--- guardrail. No-op in the tester/optimizer (m_inferenceOnly) and while training is active.
void OnlineLearnStep(void);
//--- Alpha-balanced focal sample weight (Lin et al. 2017 eq. 5) for ONE streamed bar - see the
//--- ONLINE_LEARN_* block's CLASS IMBALANCE comment for the derivation. Shared deliberately by
//--- OnlineLearnStep() (the live path) and AdvanceOosSimulationChunk() (the simulation of that
//--- path): the simulation's reported accuracy is only a valid forecast of live continual-learning
//--- behaviour if it optimises the IDENTICAL objective, so the formula must exist in exactly one
//--- place. p* are this bar's pre-update softmax probabilities, already normalised in place by
//--- ApplyClassificationSoftmax(). Returns 1.0 for the regression head (no class structure).
double OnlineSampleWeight(ENUM_SIGNAL trueSignal, double pBuy, double pSell, double pNeutral);
//--- lazily bootstraps m_shadowNet if it's still NULL: tries loading a persisted shadow file
//--- first (continuity across EA restarts), falling back to cloning Net's current weights (via
//--- the same Save()/Load() pattern StartOosContinualSimulation() uses for m_simOosNet) if no
//--- compatible shadow file exists yet. No-op if m_shadowNet is already valid. See m_shadowNet's
//--- declaration comment for the full EMA shadow-weight deployment rationale.
void EnsureShadowNet(void);
//--- persists m_shadowNet alongside every Net.Save() call, using the same run metadata (error/
//--- undefine/forecast/era/trainingComplete/indicator params) the caller already computed for
//--- Net.Save() itself - see m_shadowNet's declaration comment. No-op if the shadow isn't
//--- bootstrapped yet.
void SaveShadowNet(const double &indicatorParams[]);
//--- method of initialization of the indicators
bool InitOpen(CIndicators *indicators);
bool InitClose(CIndicators *indicators);
bool InitHigh(CIndicators *indicators);
bool InitLow(CIndicators *indicators);
bool InitVolumes(CIndicators *indicators);
bool InitTime(CIndicators *indicators);
//--- addToCollection=false is used by ReInitADIndicators() to rebuild an already-collected
//--- handle's params (here: a re-tuned period) without re-adding the (same) pointer into
//--- indicators a second time
bool InitMA(CIndicators *indicators, bool addToCollection = true);
bool InitRSI(CIndicators *indicators, bool addToCollection = true);
bool InitMACDFeature(CIndicators *indicators, bool addToCollection = true);
bool InitIchimoku(CIndicators *indicators, bool addToCollection = true);
bool InitADCumulativeDelta(CIndicators *indicators, bool addToCollection = true);
bool InitADShorteningOfThrust(CIndicators *indicators, bool addToCollection = true);
bool InitADWyckoffEventStream(CIndicators *indicators, bool addToCollection = true);
bool InitADWyckoffFailedStructure(CIndicators *indicators, bool addToCollection = true);
bool InitADWyckoffSignificantBarInversion(CIndicators *indicators, bool addToCollection = true);
bool InitADZigZag(CIndicators *indicators, bool addToCollection = true);
//--- common=false targets a LOCAL (non-shared) file - used by the tester/optimizer per-agent
//--- weight cache so cross-pass reuse never touches the production FILE_COMMON config/weights.
feat(nn): derive dense depth, train on all history, pin the shape in .cfg Completes the derived-topology work. Three inputs removed. AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five entries instead of eight. Depth is now derived from the two endpoints the taper already has to connect (derived first-layer width, output-tied final width) at a 2x per-layer compression target, clamped [2..5]. Asking a user to pick a layer count while the code derives the widths those layers taper between was asking for half a decision: at 64 units tapering to 12, four layers compress by 1.4x per step and five by 1.3x, so the extra depth bought no abstraction. On the shipping H1/10y default the derivation lands on 3 layers - the depth that actually won Run 2. StudyPeriods removed. There is no case for training on less data than the broker provides at a ~6% directional base rate; the honest generalization read comes from the OOS holdout, not from withholding history. Training now starts at the earliest available bar, floored by MinTrainYear, which answers a different question (excluding dubious pre-history) and stays. That required closing the hazard the old code documented: the capacity budget now MEASURES the symbol's real bar count, and a topology derived from a measurement would widen as history downloads. Both ends are now pinned. Every derived value left the weights-filename fingerprint - keying a filename on a measured quantity means the EA looks for a file that does not exist, starts from era 0 and orphans a trained model, silently, because a missing cache is the normal first-run state. The shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the four derived fields rather than diffing them; a mismatch there would discard a fully-trained model over nothing the user did. Two fields appended to the .cfg for the conv/LSTM stages, length-guarded on read because FileReadInteger past EOF returns 0 with no error. ForceHiddenLayers, a compile-time constant like DebuggingMode, pins depth for diagnostic comparisons. It joins the fingerprint only when non-zero, so forced depths get their own files - sequential comparisons only, not simultaneous from one .ex5. Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64, 3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from ~58k to ~28k weights. Both builds compile 0 errors, 0 warnings. Re-keys existing models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
bool SaveTopologyConfiguration(string fileName, int initialNeuronsCount, int hiddenLayersCount, double neuronsReduction, int minNeuronsCount, int optimizationAlgo, int historyBars, int outputNeuronsCount, int neuronsCount, int studyPeriod, int minTrainYear, bool isInitialized, int stopTrainWR, int fractalPeriods, int convFilterCount, int lstmHiddenSize, bool common = true);
//--- The four DERIVED shape fields are by REFERENCE and are ADOPTED from the .cfg, not compared
//--- against it. See the block in the definition for why a derived value must never be able to
//--- mismatch: it is measured from data that legitimately changes, and a mismatch here discards
//--- a trained model. studyPeriod left the parameter list entirely - the input is gone; its
//--- on-disk slot is still read positionally and ignored, like the retired MinWR slot.
bool LoadAndCompareTopologyConfiguration(string fileName, int &initialNeuronsCount, int &hiddenLayersCount, double neuronsReduction, int minNeuronsCount, int optimizationAlgo, int &historyBars, int outputNeuronsCount, int neuronsCount, int minTrainYear, bool isInitialized, int stopTrainWR, int fractalPeriods, int &convFilterCount, int &lstmHiddenSize, bool common = true);
//--- Retry helpers for the tester/opt seed-copy race: a live chart's own atomic Save() (write .savetmp,
//--- then FileMove() over the real file) can hold the source or destination file for a moment, and a
//--- concurrent FileCopy/FileOpen from a Strategy Tester agent reading the SAME production file can hit
//--- a transient Windows sharing violation in that narrow window. Both retry a handful of times with a
//--- short pause rather than silently treating a transient lock as "no model"/"corrupt file" - see their
//--- call sites in InitNeuralNetwork.
bool CopyFileWithRetry(string srcFileName, string dstFileName);
bool CopySharedFile(string srcFileName, string dstFileName, bool quiet);
bool LoadNetWithRetry(double &indicatorParams[]);
//--- input data
bool m_useVolumes;
bool m_useTime;
bool m_useATR;
//--- Uses its own period (m_indicatorTuner.maPeriod), fed as ATR-normalized OHLC distance-from-MA
//--- (4 values, same convention as the base close-open/high-open/low-open features) plus the MA's
//--- own bar-over-bar change (1 value, ATR-normalized like every other price-domain feature here -
//--- not volume's previous-bar-ratio scheme, since a moving average lives in price units and
//--- already has ATR as its natural scale reference). See BufferTempDataCompute()'s m_useMA block
//--- for the exact 5 values. maPeriod starts equal to the Classic Signals PeriodMA input (see
//--- CADIndicatorTuner's constructor) but may diverge from it once AutoTuneIndicators searches a
//--- trial - the Classic Signals MA vote itself is untouched by that search, since it needs no
//--- training/warm-up and there is nothing for a tuning trial to validate it against.
bool m_useMA;
//--- RSI is already a 0-100 oscillator, so the only transform needed is /100 to match every other
//--- feature's roughly [-1,1]/[0,1] scale - no ATR or distance normalization applies. See
//--- BufferTempDataCompute()'s m_useRSI block for the exact value. Same maPeriod/rsiPeriod
//--- divergence-from-the-Classic-Signals-input note as m_useMA above applies to rsiPeriod.
bool m_useRSI;
//--- MACD as 3 ATR-normalized values (main line, signal line, histogram) - see
//--- BufferTempDataCompute()'s m_useMACD block. Deliberately kept to 3 despite MACD being cheap: the
//--- point of adding it next to m_useMA is the SECOND timescale (m_useMA supplies exactly one moving
//--- average) and the histogram, which is the only acceleration term anywhere in the feature vector -
//--- the level information itself is already covered by the MA distances. ATR-normalized rather than
//--- left raw because the MACD lines live in price units, exactly like the MA feature.
bool m_useMACD;
//--- Ichimoku as 8 values - see BufferTempDataCompute()'s m_useIchimoku block for each. This is the
//--- widest single classic-indicator feature here and it earns that width by carrying multi-timescale
//--- support/resistance GEOMETRY (three lookbacks plus a forward-projected cloud) that nothing else in
//--- the vector encodes: the swing-context block's confirmed pivots are >=m_swingConfirmationBars bars
//--- stale by construction, and its Donchian/SMA values are single-scale.
//--- LOOKAHEAD - MT5's iIchimoku stores raw per-bar values and shifts only the DRAWING, so the cloud
//--- sitting under bar idx is SenkouSpan*(idx + ichiKijun), and the Chikou value plotted at bar idx
//--- would be Close(idx - ichiKijun) - a FUTURE bar. The feature block applies the +Kijun offset and
//--- never calls ChinkouSpan(); Signals\SignalIchimoku.mqh's class comment documents the buffer
//--- convention in full, and the same reasoning governs both.
bool m_useIchimoku;
//--- Normalized ZigZag swing-context features: 5 confirmed-pivot values (direction/magnitude/age of
//--- the last CONFIRMED swing) plus 4 recent-price-action values (Donchian range position at 20/50
//--- bars, 20-bar return, 20-bar SMA extension) that give fresh, non-repainting trend/position
//--- context the >=100-bar-stale pivot anchor can't - see BufferTempDataCompute()'s m_useSwingContext
//--- block for the exact 9 values. Reads the same
//--- m_ADZigZag the training labels already come from (see that member's declaration comment) rather
//--- than a separate indicator instance - NOT gated behind AutoTuneIndicators/ReInitADIndicators like
//--- the AD* indicators below, since m_ADZigZag itself is deliberately never tuned (same reason as
//--- the label side: tuning the ground truth alongside the model scored against it would let a trial
//--- cherry-pick an easier target). Critical correctness constraint, not just a style choice: ZigZag's
//--- most recent 1-3 legs repaint (see m_swingConfirmationBars' declaration comment), so every read of
//--- m_ADZigZag for THIS feature - exactly like the label side - must only trust a pivot that is
//--- already at least m_swingConfirmationBars bars old relative to the bar the feature is being
//--- computed for. Skipping that embargo would leak future information a live bar couldn't actually
//--- have had yet - silent lookahead bias inflating backtest/training performance without being real.
bool m_useSwingContext;
//--- see System\NewsRelevance.mqh's declaration comment for what this feature actually encodes
//--- (event proximity + impact, not actual-vs-forecast deviation) and why the forward-looking half
//--- of it isn't lookahead bias.
bool m_useNews;
int m_newsFeatureWindowMinutes;
fix(signals): revive a dead MA model, and demote Sanyaku from state to event Two defects surfaced by research/test_classic.py, both verified fixed by re-running the transcription against 178k bars of EURUSD H1. CSignalMA model 1 could never fire. For any recursive average - and MA_TYPE_EMA is the shipped default - MA(i) = a*Close(i) + (1-a)*MA(i+1), so DiffMA(i) = a * (Close(i) - MA(i+1)) DiffCloseMA(i) = (1-a) * (Close(i) - MA(i+1)) are positive multiples of one quantity and always share a sign. Model 1 asks for a close BELOW a RISING average, which is precisely the combination that identity forbids: 0.000% of bars, either direction, any symbol. The MQL5 standard library this was ported from defaults to MODE_SMA, where the two are merely correlated - the bug arrived with the EMA default, not with the port. Reading the slope one bar back (DiffMAPrev) breaks the tie for every MA type while keeping the model's stated meaning. Now fires on 7.92% of bars. CSignalIchimoku model 11 fired on 27% of bars at weight 100. Sanyaku is three standing STATES conjoined with no transition term, so it held across long stretches - and being last in the if-chain at the top weight, the module's highest-conviction reading was also its most common one, overwriting all eight event models below it on a quarter of all bars. The old comment rejected an event form because "demanding all three flip on the same bar would fire almost never" - true, but that is not the alternative. Kouten is the TURN: the ALIGNMENT transitions, and only one role need change for it to. Testing !Sanyaku(idx+1) fires once per aligned stretch. Now 2.17%, in line with Kumo breakout (2.4%) and the strong TK cross (1.1%). DataReady() extended one bar deeper to cover the lookback. Neither pattern showed edge before or after; this is about the models meaning what they say and the vote not being dominated by a constant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:14:34 -04:00
//--- Cross-asset panel: the only feature block here whose inputs are NOT a transform of this
//--- symbol's own OHLCV series. See System\CrossAsset.mqh for the reasoning; in short, every other
//--- feature the network sees is a function of one price series, and that whole family measured at
//--- the noise floor, so the panel exists to give it information a single series cannot contain.
//--- Built ONCE per training run (BuildCrossAssetPanel) rather than per bar - a per-bar cross-symbol
//--- lookup would be pairs x bars iBarShift calls.
bool m_useCrossAsset;
CCrossAssetPanel m_crossAsset;
bool BuildCrossAssetPanel(int bars);
//--- Train->serve parity for the panel (2026-08-11): the pair set is a MEASURED property of the
//--- terminal, so like the derived barrier pair it is pinned in the .cfg, not the filename hash
//--- (see BuildConfigFingerprint's XA note). Empty = no set pinned yet; first successful Build
//--- stamps it and re-saves the .cfg one-shot, exactly the m_geometryCfgSaved pattern.
string m_crossAssetPairsPinned;
bool m_crossAssetCfgSaved;
feat(ai): spread as a volatility-regime feature, and fix a stale-index cache in both new blocks Adds spread/ATR and the spread change ratio as network inputs (EnableSpreadFeature, default on). Spread is the one microstructure channel that is both FX-available and genuinely historical in the Strategy Tester - "during testing, the spread is not modeled but is taken from historical data" - so unlike swap, signed tick flow or depth of market it is something a backtest can honestly validate. What it encodes, stated precisely because the raw measurement overstates it. research/test_spread.py found spr/atr the strongest single feature in this codebase, on 5 of 8 instrument/geometry cells at 2-4x any volume feature. But the barrier LABEL charges the spread inside its own barriers, so a wide-spread bar is mechanically likelier to resolve as a loss and the feature would partly be predicting its own cost model. Relabelling at zero cost and re-measuring the identical feature showed 20-40% of it WAS that tautology and the majority was not (XAUUSD retained 97%). What survives is a volatility-regime reading: spread is near-fixed while ATR is not, so the ratio runs high exactly when realised volatility is below its own ATR estimate, which genuinely predicts whether ATR-scaled barriers get reached. It is UNSIGNED - Neutral-vs-directional only, never a side. Also fixes a stale-index bug I introduced with the cross-asset panel and had just repeated in the spread series. Both cached on length alone: if(m_crossAsset.Bars() >= bars) return true; MQL5 series indices are relative to NOW, so one new closed candle shifts every index by one. Keyed only on length, the panel keeps serving its index 0 as a bar that is no longer the newest, and every cross-asset value is read one bar out of step with the price features sitting beside it in the same vector - silently, with no error and no shape change. This is the same class of defect as the dtStudied watermark behind the zero-direction backtests. Both now carry a datetime anchor on m_Time.GetData(0), the same invalidation key the label/feature bar caches already use. And a performance fix that fell out of it: with correct invalidation the panel rebuilds on every new bar, and RefreshConvergedSignal runs per bar - which in the tester would mean one full multi-symbol resample per simulated bar at training depth. Inference only reads bars 0..m_historyBars-1 plus the panel's own slow window, so it now requests exactly that. The cache check is >=, so a deeper panel left from training still satisfies it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:42:40 -04:00
//--- Spread as a feature. The only microstructure channel that is BOTH available on FX and
//--- genuinely historical in the Strategy Tester ("during testing, the spread is not modeled but
//--- is taken from historical data") - unlike swap (no history at all), signed tick flow
//--- (TICK_FLAG_BUY/SELL are empty on FX) or depth of market (absent on retail FX, never replayed).
//--- Measured as the strongest single feature in research/test_spread.py, though see the feature
//--- block for what it actually encodes and why that is less than it first appears.
//--- Series is copied ONCE per bar grid, not per bar: CopySpread is a range call, not a lookup.
bool m_useSpreadFeature;
int m_spreadSeries[];
int m_spreadSeriesBars;
//--- Newest bar the copy was anchored to. MQL5 series indices are relative to NOW, so a single
//--- new closed candle shifts every index by one: a cache keyed only on length would keep serving
//--- index 0 as a bar that is no longer the newest, silently misaligning the spread series against
//--- the price buffers it must line up with. Same invalidation key the label/feature bar caches
//--- use (see EnsureBarCachesCapacity) and the same failure the zero-direction hunt traced.
datetime m_spreadSeriesAnchor;
datetime m_crossAssetAnchor;
bool EnsureSpreadSeries(int bars);
bool m_useADCumulativeDelta;
bool m_useADShorteningOfThrust;
bool m_useADWyckoffEventStream;
bool m_useADWyckoffFailedStructure;
bool m_useADWyckoffSignificantBarInversion;
public:
CExpertSignalAIBase(void);
~CExpertSignalAIBase(void);
//--- "voting" that price will grow/fall, common to every AI signal (single market model)
virtual int LongCondition(void);
virtual int ShortCondition(void);
// |dPrevSignal| is already a 0..1 confidence for classification output (softmax
// probability of the winning class) and typically bounded for regression output
// (tanh-activated network); OpenParams() clamps regardless. Scaled by m_confidenceCalScale
// (classification head only - 1.0/no-op for regression) so callers get an empirically
// calibrated magnitude instead of the raw, uncalibrated softmax value - see
// m_confidenceCalScale's declaration comment.
double CalibratedConfidenceMagnitude(void) const
{
double mag = MathAbs(dPrevSignal);
if(!MathIsValidNumber(mag))
return 0.0;
if(m_outputNeuronsCount == 3)
mag = MathMin(1.0, mag * m_confidenceCalScale);
if(!MathIsValidNumber(mag))
return 0.0;
return mag;
}
virtual double AIConfidence(void) override { return CalibratedConfidenceMagnitude(); }
// Signed for direction-aware use (AI-driven early exit): sign matches dPrevSignal's
// convention (+ buy, - sell, 0 neutral/no signal yet). dPrevSignal == -2 is the
// "not yet studied" sentinel, not a real sell signal - treat it as no confidence.
virtual double SignedAIConfidence(void) override
{
if(dPrevSignal == -2)
return 0.0;
double sign = (dPrevSignal > 0.0) ? 1.0 : (dPrevSignal < 0.0) ? -1.0 : 0.0;
if(sign == 0.0)
return 0.0;
return sign * CalibratedConfidenceMagnitude();
}
//--- event handlers, common to every AI signal
virtual void OnTickHandler(void);
//--- drives the same training-scheduling check as OnTickHandler(), but callable from a timer so
//--- it isn't dependent on ticks (which don't arrive while the market is closed)
void PollTraining(void);
virtual void OnChartEventHandler(const int id,
const long &lparam,
const double &dparam,
const string &sparam);
//--- methods of adjusting "weights" of the 4 confidence-tier market models - see m_pattern_0's
//--- declaration comment
void Pattern_0(int value) { m_pattern_0 = value; }
void Pattern_1(int value) { m_pattern_1 = value; }
void Pattern_2(int value) { m_pattern_2 = value; }
void Pattern_3(int value) { m_pattern_3 = value; }
virtual void ApplyPatternWeight(int patternNumber, int weight);
//--- buckets the live confidence magnitude into one of the 4 tiers above - see m_pattern_0's
//--- declaration comment. Public so PollTraining()/status-display code could surface which tier is
//--- currently active if ever useful, though LongCondition/ShortCondition are the only callers today.
int ConfidenceTier(void);
int PatternWeightForTier(int tier);
//--- methods of setting adjustable parameters
refactor(ai): derive the first dense layer's width instead of asking for it InitialNeurons was an input whose only defensible value depends on two things the user cannot see when picking from a dropdown: how wide the input vector ended up after feature selection, and how much in-sample data the study period actually yields. Left to a hand-picked constant it was badly wrong - 500 units against a 420-wide input is 210,500 weights, 72% of a 292,583-weight model, against ~36,500 training bars of which only ~2,236 are directional. That is 6.6 weights per training bar, and it EXPANDS a set of highly correlated inputs rather than compressing them. The symptom was already in the logs and had been read as a depth problem: the shallowest topology consistently beat the deepest (perceptron 52.7% balanced, hybrid 41.3%). Over-parameterization predicts that ordering just as well as covariate shift does, and only one of the two had been addressed. ComputeFirstLayerWidth() budgets roughly one first-layer weight per in-sample bar. Measured across the configurations in use: M15 10y -> 256 units, 129,071 weights, 0.73 per bar H1 10y -> 64 units, 28,727 weights, 0.65 per bar H4 10y -> 16 units, 7,559 weights, 0.68 per bar Two design points that matter: - It estimates in-sample bars from the STUDY PERIOD and timeframe, not from Bars(). What is downloaded grows over a terminal's lifetime, and a topology that widened as history filled in would re-key its own weights file and discard a trained model. - The result is snapped down to a coarse power-of-two ladder, so the estimate would have to be wrong by ~2x to change the answer. Every field it reads is already part of the weights-filename fingerprint, so the derived value needs no fingerprint entry of its own. The public setter is removed - it could only have been called after construction, and would either be ignored or silently re-key the model mid-run. Where the data cannot support even the floor (D1 over 10 years is under 2,000 bars) it now says so and names the fixes, rather than quietly training a model with more weights than examples. The DB config fingerprint drops the term too, which re-keys existing pattern databases once - correct, since a model an order of magnitude smaller should not inherit the old one's win-rate history. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 13:01:16 -04:00
//--- No public setter for m_initialNeuronsCount. It feeds the weights-filename fingerprint, and it is
//--- now derived exactly once, inside InitNeuralNetwork(), before that fingerprint is built - see
//--- ComputeFirstLayerWidth(). An external setter could only ever be called after construction and
//--- would either be ignored (if before init) or silently re-key the model mid-run (if after).
void OutputNeuronsCount(int value) { m_outputNeuronsCount = value; }
refactor(ai): derive the dense taper's shape, not just its first layer Deriving the first layer's width left NeuronsReduction and MinNeuronsCount behind as inputs calibrated for something that no longer exists. Against a hand-picked 500-wide first layer "keep 30%, floor at 20" produced a genuine funnel - 500 -> 150 -> 45. Against the derived 64 it degenerates to 64 -> 20 -> 20: the reduction factor stops mattering after one step, and "minimum neurons per layer" silently becomes the width of every layer but the first. Two knobs whose labels no longer describe what they do. The taper now runs geometrically from the derived first-layer width down to a final hidden layer sized off the output count, spread evenly over however many layers the chosen AIType implies: MLP_3L 64 -> 28 -> 12 -> 3 29,151 dense weights MLP_4L 64 -> 37 -> 21 -> 12 -> 3 30,450 CONV/LSTM/HYBRID_2L 64 -> 12 -> 3 27,763 and it stays a funnel at the floor, where the old rule could not: D1 (first layer floored to 16) 16 -> 14 -> 12 -> 3 Both inputs are removed. With the width derived there is no freedom left in the taper, so keeping either would only let the user contradict the derivation. The layer COUNT stays selectable, because it is bundled into AIType alongside the conv/LSTM front-end - depth is an architecture choice, not a data-derived quantity, and pairing them means the two cannot contradict each other. m_minNeuronsCount / m_neuronsReduction survive as frozen members: nothing reads them to build a topology any more, but they hold positional slots in the .cfg sidecar and the weights fingerprint, and changing either value would re-key every model on disk for no behavioural reason. The DB config fingerprint drops both terms. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 14:03:42 -04:00
//--- No setter: the taper's endpoints are derived, not configured. See BuildFreshTopology()'s taper
//--- block and the note in Variables\Inputs.mqh. Kept as members only because the .cfg topology
//--- sidecar's field layout is positional and rewriting it would invalidate every model on disk.
void HiddenLayersCount(int value) { m_hiddenLayersCount = value; }
void LstmHiddenSize(int value) { m_lstmHiddenSize = value; }
void ConvFilterCount(int value) { m_convFilterCount = value; }
//--- StopTrainWR(int) removed with the MinWR input - there is no absolute accuracy target any more;
//--- see LEGACY_CONVERGE_WR_SLOT and the plateau ladder.
void MinDirectionalRecall(int value) { m_minDirectionalRecallPct = value; }
//--- MinSignalConfidence(double) removed with the AI entry floor - confidence now reaches the trade
//--- decision as vote weight (ConfidenceTier), gated by the one Min vote to open threshold that the
//--- classic votes already answer to. See m_pattern_0's declaration comment.
refactor(ai): nine class-imbalance inputs down to two The imbalance section offered nine controls for one job. Audited against the code, five of them did not do what their names said at the shipped defaults: AILogitPriorStrength DEAD - Inference.mqh's post-hoc prior early-returns whenever the adjusted loss is on, which is default. OversampleParity DEAD in training - Training.mqh gated the replay loop on !useLogitAdjustedLoss (correctly, citing Buda et al. 2018). Live only in the online-learning path. EnableMinorityReplay DEAD as replay. It survived ONLY as a focal-gamma damper - "replay minority bars through pass-2 oversampling" was a focal-loss switch. ConstrainReplay DEAD as a cap; it only chose damper 0.125 vs 0.25. UseStaticPrior An exact duplicate of FreezePriorCalibration - the two were OR'd together in the single place either is read. So they were not five mechanisms fighting; they were one mechanism plus eight knobs that mostly described machinery that no longer ran. That is worse than a real conflict, because the log agreed with the names: the label-cache line printed "reps up to 28x (90% parity) (seeding era 0's class-balance oversampling)" on every run, describing an oversampling pass that had been switched off. It is fixed here too - it cost this session a wrong diagnosis. The one genuine redundancy was focal loss, running at gamma*0.125 alongside the adjusted loss: two corrections on the same axis, the exact stacking failure this file already cited Buda et al. for in two other places, damped by a replay flag whose replay path was itself dead. Removed rather than re-tuned. The plateau ladder is unaffected - its escape is the learning-rate warm restart; the gamma anneal beside it only ever stepped toward zero. WHAT REMAINS is logit-adjusted loss (Menon et al. 2021) plus a prior freeze: LogitAdjustTau 0 = off; replaces the separate EnableLogitAdjusted- Loss boolean, since a strength dial where 0 already means off does not need an on/off switch beside it. FreezePriorCalibration unchanged. It is the only one of the six corrections with a consistency guarantee, and it is consistent for exactly the balanced-error metric checkpoint selection already ranks on - so the loss and the deploy decision optimize one thing. The online continual-learning path keeps its own alpha-balanced focal weight, now as constants pinned to the removed inputs' shipped defaults, so its behaviour is unchanged. It legitimately needs its own correction: ApplyLogitAdjustment() only runs inside a training run, so a deployed model that was reloaded carries no logit offsets and would otherwise stream 31:1 data into itself uncorrected. The weights-filename fingerprint is BYTE-IDENTICAL. The focal slot was a double fed to a %d conversion and had always emitted a literal 0; the |MR: segment is written as the constant its shipped defaults produced. Dropping either would have re-keyed every model and forced a from-scratch retrain of the one topology currently converged and trading. Also removed as orphans: FOCAL_GAMMA_PRESET, MAX_OVERSAMPLE_REPLICAS, OVERSAMPLE_PARITY_FRACTION, PLATEAU_GAMMA_STEP, and the now-unreachable "neutralized by prior correction" diagnostic. Both builds compile 0 errors, 0 warnings. No retrain forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 11:46:57 -04:00
//--- The SINGLE class-imbalance control - tau in Menon et al.'s logit adjustment. 0 disables the
//--- correction entirely. OversampleParity/EnableMinorityReplay/ConstrainReplay/LogitPriorStrength/
//--- UseLogitAdjustedLoss/FocalLossGamma/UseStaticPrior were removed 2026-07-31; see the
//--- class-imbalance block in Variables\Inputs.mqh for the audit that found five of them inert.
feat(ai): logit-adjusted loss, replacing oversampling and the post-hoc prior Menon et al. 2021 (ICLR), "Long-tail learning via logit adjustment": add tau*log(prior_c) to each class logit inside the training gradient. Softmax CE on adjusted logits is consistent for BALANCED error - the metric checkpoint selection already ranks on - so the loss and the deploy decision finally optimize the same thing. The engine already computed a true softmax + categorical-CE gradient and wrote it over the per-neuron sigmoid delta, so this is an offset added to three logits in the two places that gradient is built (backProp scalar path and backPropOCL). No backend, kernel or DLL change; the forward pass and every inference path are untouched, which is the point - the network learns to absorb the offset, so its raw argmax becomes the balanced-optimal decision with nothing applied at inference. Replaces rather than stacks. Minority replay is disabled while this is on, and the post-hoc inference prior is forced off. Stacking is not a theoretical worry: simulated on the measured 1118/1119/34298 distribution in the weak-signal regime, plain CE collapses to Neutral (33.4% balanced, Buy 0%), replay reaches 48.1%, logit adjustment 50.9% with better balance - and BOTH together score 45.4% with Neutral recall at 0%, worse than either alone. Buda et al. 2018 predicts exactly that. Motivation from the six-chart run: every topology took one direction to ~50% recall and abandoned the other, the direction chosen arbitrarily (the batch-norm control went Buy 1% / Sell 42%, the inverse of the other five). One era in 1,301 cleared the per-class recall floor. Fingerprinted conditionally, so the converged 60.7% models on disk keep their filenames and stay loadable as the fallback. Both builds compile 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:05:14 -04:00
void LogitAdjustTau(double value) { m_logitAdjustTau = MathMax(0.0, value); }
void FreezePriorCalibration(bool value) { m_freezePriorCalibration = value; }
void SignalClusterWindow(int value) { m_signalClusterWindow = value; }
void SwingConfirmationBars(int value) { m_swingConfirmationBars = value; }
void EnableOnlineLearning(bool value) { m_enableOnlineLearning = value; }
void MaxErasPerRun(int value) { m_maxErasPerRun = value; }
void OOSSplit(int value) { m_oosSplitPct = value; }
void HistoryBars(int value) { m_historyBars = value; }
void MinTrainYear(int value) { m_minTrainYear = value; }
void UseVolumes(bool value) { m_useVolumes = value; }
void UseTime(bool value) { m_useTime = value; }
void UseATR(bool value) { m_useATR = value; }
void UseMA(bool value) { m_useMA = value; }
void UseRSI(bool value) { m_useRSI = value; }
void UseMACD(bool value) { m_useMACD = value; }
void UseIchimoku(bool value) { m_useIchimoku = value; }
void UseSwingContext(bool value) { m_useSwingContext = value; }
void UseNews(bool value) { m_useNews = value; }
void NewsFeatureWindowMinutes(int value) { m_newsFeatureWindowMinutes = value; }
fix(signals): revive a dead MA model, and demote Sanyaku from state to event Two defects surfaced by research/test_classic.py, both verified fixed by re-running the transcription against 178k bars of EURUSD H1. CSignalMA model 1 could never fire. For any recursive average - and MA_TYPE_EMA is the shipped default - MA(i) = a*Close(i) + (1-a)*MA(i+1), so DiffMA(i) = a * (Close(i) - MA(i+1)) DiffCloseMA(i) = (1-a) * (Close(i) - MA(i+1)) are positive multiples of one quantity and always share a sign. Model 1 asks for a close BELOW a RISING average, which is precisely the combination that identity forbids: 0.000% of bars, either direction, any symbol. The MQL5 standard library this was ported from defaults to MODE_SMA, where the two are merely correlated - the bug arrived with the EMA default, not with the port. Reading the slope one bar back (DiffMAPrev) breaks the tie for every MA type while keeping the model's stated meaning. Now fires on 7.92% of bars. CSignalIchimoku model 11 fired on 27% of bars at weight 100. Sanyaku is three standing STATES conjoined with no transition term, so it held across long stretches - and being last in the if-chain at the top weight, the module's highest-conviction reading was also its most common one, overwriting all eight event models below it on a quarter of all bars. The old comment rejected an event form because "demanding all three flip on the same bar would fire almost never" - true, but that is not the alternative. Kouten is the TURN: the ALIGNMENT transitions, and only one role need change for it to. Testing !Sanyaku(idx+1) fires once per aligned stretch. Now 2.17%, in line with Kumo breakout (2.4%) and the strong TK cross (1.1%). DataReady() extended one bar deeper to cover the lookback. Neither pattern showed edge before or after; this is about the models meaning what they say and the vote not being dominated by a constant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:14:34 -04:00
void UseCrossAsset(bool value) { m_useCrossAsset = value; }
feat(ai): spread as a volatility-regime feature, and fix a stale-index cache in both new blocks Adds spread/ATR and the spread change ratio as network inputs (EnableSpreadFeature, default on). Spread is the one microstructure channel that is both FX-available and genuinely historical in the Strategy Tester - "during testing, the spread is not modeled but is taken from historical data" - so unlike swap, signed tick flow or depth of market it is something a backtest can honestly validate. What it encodes, stated precisely because the raw measurement overstates it. research/test_spread.py found spr/atr the strongest single feature in this codebase, on 5 of 8 instrument/geometry cells at 2-4x any volume feature. But the barrier LABEL charges the spread inside its own barriers, so a wide-spread bar is mechanically likelier to resolve as a loss and the feature would partly be predicting its own cost model. Relabelling at zero cost and re-measuring the identical feature showed 20-40% of it WAS that tautology and the majority was not (XAUUSD retained 97%). What survives is a volatility-regime reading: spread is near-fixed while ATR is not, so the ratio runs high exactly when realised volatility is below its own ATR estimate, which genuinely predicts whether ATR-scaled barriers get reached. It is UNSIGNED - Neutral-vs-directional only, never a side. Also fixes a stale-index bug I introduced with the cross-asset panel and had just repeated in the spread series. Both cached on length alone: if(m_crossAsset.Bars() >= bars) return true; MQL5 series indices are relative to NOW, so one new closed candle shifts every index by one. Keyed only on length, the panel keeps serving its index 0 as a bar that is no longer the newest, and every cross-asset value is read one bar out of step with the price features sitting beside it in the same vector - silently, with no error and no shape change. This is the same class of defect as the dtStudied watermark behind the zero-direction backtests. Both now carry a datetime anchor on m_Time.GetData(0), the same invalidation key the label/feature bar caches already use. And a performance fix that fell out of it: with correct invalidation the panel rebuilds on every new bar, and RefreshConvergedSignal runs per bar - which in the tester would mean one full multi-symbol resample per simulated bar at training depth. Inference only reads bars 0..m_historyBars-1 plus the panel's own slow window, so it now requests exactly that. The cache check is >=, so a deeper panel left from training still satisfies it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 17:42:40 -04:00
void UseSpreadFeature(bool value) { m_useSpreadFeature = value; }
void UseADCumulativeDelta(bool value) { m_useADCumulativeDelta = value; }
void UseADShorteningOfThrust(bool value) { m_useADShorteningOfThrust = value; }
void UseADWyckoffEventStream(bool value) { m_useADWyckoffEventStream = value; }
void UseADWyckoffFailedStructure(bool value) { m_useADWyckoffFailedStructure = value; }
void UseADWyckoffSignificantBarInversion(bool value) { m_useADWyckoffSignificantBarInversion = value; }
void AutoTuneIndicators(bool value) { m_autoTuneIndicators = value; }
//--- control-panel API (Warrior_EA.mq5): current-config-only training/weights control.
//--- "current config" == this signal instance's own m_fileName (symbol+period+id+topology),
//--- never touches another signal type's or another symbol/timeframe's saved files.
void PauseTraining(void) { m_trainingPaused = true; PrintVerbose(ID + ": training paused by user (era " + IntegerToString(m_eraCount) + ")"); }
void ResumeTraining(void) { m_trainingPaused = false; PrintVerbose(ID + ": training resumed by user (era " + IntegerToString(m_eraCount) + ")"); }
bool IsTrainingPaused(void) const { return m_trainingPaused; }
bool IsTrainingStopped(void) const { return m_trainingStopRequested; }
bool TrainingComplete(void) const { return m_trainingComplete; }
fix(deinit): a full model write was running ahead of the cheap cleanup "Abnormal termination" is back, and this time it is not the arrows. The timing names the culprit exactly: 16:02:31.547 OnDeinit: shutting down 16:02:36.003 Abnormal termination <- 4.46 s, MetaTrader gave up 16:02:36.226 chart signals - persisted <- cleanup finished 0.2 s LATE OnDeinit called StopTraining() BEFORE the chart cleanup. StopTraining() finalises an in-flight run, and FinalizeTrainRun() restores the best checkpoint and then persists it - a full ~1MB model write per signal. So the expensive step ran ahead of the cheap bounded one, which is precisely the inversion the shutdown ordering exists to prevent. The previous fix put PersistWeightsOnShutdown last and missed that StopTraining smuggles a second save in at the front. Two changes: Cleanup now runs FIRST, then StopTraining, then the weight save. The visible teardown is cheap and bounded, so it always completes even when everything after it is killed. And the deploy-persist inside FinalizeTrainRun is suppressed during shutdown. RestoreWeights() is an in-MEMORY swap, so the best checkpoint is already the live net by that line, and PersistWeightsOnShutdown writes exactly those weights moments later. The old path wrote the same model twice per signal - eight full writes across four charts - for no benefit. A user-pressed Stop still persists immediately, because nothing else would. Compiles 0 errors / 0 warnings. Build tag deinit-order-v2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 16:06:40 -04:00
//--- Set by OnDeinit before it calls StopTraining(), so FinalizeTrainRun() can tell a user-pressed Stop
//--- (persist the deployed model now - nothing else will) from a shutdown (PersistWeightsOnShutdown is
//--- moments away and writes the same bytes). See the guard in FinalizeTrainRun.
void MarkShutdown(void) { m_shutdownInProgress = true; }
fix: flush the in-flight era on shutdown; sweep orphaned chart objects on attach Chart objects live in the MT5 chart PROFILE, not in this EA's files. They survive a terminal restart, a recompile, and deleting every .nnw/.cfg/.stats/.arrows on disk. Only a deinit that RUNS TO COMPLETION removes them - and MetaTrader force-terminates OnDeinit at roughly 4,500 ms, so a run killed mid-cleanup orphans them permanently with no owner left to clean up after. That is the "deleted every file, recompiled, restarted, old arrows and a stale panel still there" report: nothing was wrong with the files and deleting them could not have helped. Both halves are fixed. STOP OVERRUNNING THE BUDGET. OnDeinit used to finalise the in-flight run (StopTraining -> FinalizeTrainRun: checkpoint restore, live-state re-seed) and then write two full nets per chart. On four charts that is the bulk of the budget, spent to preserve a PARTIAL era that was never scored, never checkpointed and never deployable. FlushTrainRun() discards it instead - drop the resumable bookkeeping, leave the net neutral (unfreeze BN, flush the batch, batch size 1), skip the save - and training resumes from the last completed era, which the era-end save and the periodic autosave have already put on disk. What is discarded is bounded by one era. A CONVERGED model keeps the old finalise-and-save path: its weights can carry online-learning updates made since the last era boundary, and for a deployed model no further era boundary is coming to persist them. MAKE CLEANUP SELF-HEALING. Every purge sat behind a branch - no model loaded, sidecar missing - so the common paths returned leaving whatever the previous instance stranded. LoadChartSignals now sweeps the arrow namespace unconditionally before restoring, so the post-init chart holds exactly what the sidecar holds whichever branch runs, and the panel gets the same treatment before Create() (CAppDialog namespaces its controls, so a killed Destroy strands the lot and the next attach draws a second panel on the corpse). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 13:08:40 -04:00
//--- SHUTDOWN FLUSH: abandon an in-flight run instead of finishing it, and resume from the last
//--- COMPLETED, already-persisted era. StopTraining() does the opposite - it finalises synchronously
//--- (restore the best checkpoint from its file, re-seed live state) and OnDeinit then writes two
//--- full nets per chart. On four charts that is the bulk of the deinit budget, spent to preserve a
//--- PARTIAL era that was never scored, never checkpointed and never deployable - and the price of
//--- overrunning is that the chart cleanup gets killed and strands its arrows and panel in the chart
//--- profile, where nothing will ever clean them up (see LoadChartSignals' orphan sweep).
//--- Every completed era is already on disk: the era-end save and the periodic autosave both persist
//--- independently, so what this discards is bounded by one era of work.
//--- Returns true if there really was an in-flight run to discard, so the caller can tell a flush
//--- from a no-op and skip the weight write only when the flush actually applied.
bool FlushTrainRun(void)
{
bool inFlight = (m_trainRunActive || m_eraResumePending || m_labelPrebuildActive || m_simOosRunActive);
m_trainingStopRequested = true;
m_trainingPaused = false;
//--- Drop the resumable bookkeeping WITHOUT calling FinalizeTrainRun: no checkpoint restore, no
//--- persist, no dtStudied advance. The next start re-derives all of it from the saved model.
m_trainRunActive = false;
m_eraResumePending = false;
m_haveOosCheckpoint = false;
m_labelPrebuildActive = false;
if(m_simOosRunActive)
{
delete m_simOosNet;
m_simOosNet = NULL;
m_simOosRunActive = false;
}
//--- Leave the net in the same neutral state FinalizeTrainRun leaves it in - a frozen batch-norm or
//--- a half-filled mini-batch must not be what a later inference path finds. Cheap, unlike the save.
if(CheckPointer(Net) != POINTER_INVALID)
{
Net.SetBatchNormFrozen(false);
Net.FlushBatch();
Net.SetBatchSize(1);
}
return inFlight;
}
void StopTraining(void)
{
m_trainingStopRequested = true;
m_trainingPaused = false;
//--- ScheduleTrainingIfNeeded() refuses to schedule another "New Bar" event while
//--- m_trainingStopRequested is set, so a run interrupted mid-chunk would otherwise never get
//--- called again to finalize (restore the best checkpoint, persist state) - do it synchronously
//--- here instead. Safe to block briefly: this is just a checkpoint file restore, not the
//--- multi-minute bar loop.
if(m_trainRunActive)
FinalizeTrainRun();
Print(ID + ": training stopped by user (era " + IntegerToString(m_eraCount) + ", weights as of last completed era retained)");
diag: inference-path census, to explain zero-trade backtests A backtest of the CONVERGED CONV model produced "Final directional result: 0.00000000" on every one of 1744 bars and therefore zero trades. Nothing in the log could separate the three candidate causes, and each needs a different fix: 1. RefreshLatestSignal never called (new-bar gate never fires) 2. called, but bailing at one of its two early returns 3. running fine, and the model genuinely answers Neutral every bar Counts all three plus the Buy/Sell/Neutral split, printed once at shutdown via StopTraining (which the tester reaches through OnDeinit). Three increments per bar against a full feedForward - not worth gating. Ruled out while writing this, so the next session does not re-derive it: - the alternation gate (m_lastNonNeutralSignal) is NOT the cause. It starts at Neutral, so a first Buy would still fire and show up as one non-zero direction. We saw zero. It IS still a live hazard for a one-sided model - CONV currently calls Buy:17% Sell:0%, and after the first Buy every later Buy is suppressed until a Sell that never comes - but it cannot explain an all-zero run. - shallow buffers do not hard-fail the feature builder: the swing-context Donchian loop breaks gracefully when it runs off loaded history. It does mean converged-path inference computes Donchian/return/SMA features over a TRUNCATED window versus training, which is a real train/inference skew worth its own fix, but it degrades features rather than zeroing them. Both builds 0/0. Diagnostic only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 18:24:32 -04:00
PrintInferenceTally();
}
//--- Inference-path census, printed at shutdown. WHY: a 2026-07-31 backtest of a CONVERGED CONV model
//--- produced "Final directional result: 0.00000000" on every one of 1744 bars and therefore ZERO
//--- trades, and nothing in the log could distinguish the three candidate causes - RefreshLatestSignal
//--- never running, running but bailing at one of its two early returns, or running fine and the model
//--- genuinely answering Neutral every time. Each implies a completely different fix. Counting is the
//--- cheapest way to tell them apart and it costs nothing per bar.
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- Records one directional decision's passage through the readiness gate in LongCondition()/
//--- ShortCondition(). Called with `directional` = "this condition's own class is what the model
//--- actually answered", so a Neutral bar counts as neither blocked nor passed - the question being
//--- measured is what happened to the calls that HAD something to say.
void NoteVoteGate(bool directional)
{
if(!directional)
return;
bool open = m_trainingComplete || (m_inferenceOnly && m_modelLoadedFromDisk);
if(m_voteGateCompleteAtFirst < 0)
{
m_voteGateCompleteAtFirst = (int)m_trainingComplete;
m_voteGateLoadedAtFirst = (int)m_modelLoadedFromDisk;
}
if(open)
m_voteGatePassed++;
else
m_voteGateBlocked++;
}
diag: inference-path census, to explain zero-trade backtests A backtest of the CONVERGED CONV model produced "Final directional result: 0.00000000" on every one of 1744 bars and therefore zero trades. Nothing in the log could separate the three candidate causes, and each needs a different fix: 1. RefreshLatestSignal never called (new-bar gate never fires) 2. called, but bailing at one of its two early returns 3. running fine, and the model genuinely answers Neutral every bar Counts all three plus the Buy/Sell/Neutral split, printed once at shutdown via StopTraining (which the tester reaches through OnDeinit). Three increments per bar against a full feedForward - not worth gating. Ruled out while writing this, so the next session does not re-derive it: - the alternation gate (m_lastNonNeutralSignal) is NOT the cause. It starts at Neutral, so a first Buy would still fire and show up as one non-zero direction. We saw zero. It IS still a live hazard for a one-sided model - CONV currently calls Buy:17% Sell:0%, and after the first Buy every later Buy is suppressed until a Sell that never comes - but it cannot explain an all-zero run. - shallow buffers do not hard-fail the feature builder: the swing-context Donchian loop breaks gracefully when it runs off loaded history. It does mean converged-path inference computes Donchian/return/SMA features over a TRUNCATED window versus training, which is a real train/inference skew worth its own fix, but it degrades features rather than zeroing them. Both builds 0/0. Diagnostic only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 18:24:32 -04:00
void PrintInferenceTally(void)
{
long attempts = m_refreshOk + m_refreshFailFeatures + m_refreshFailShort;
if(attempts <= 0)
{
Print(ID + ": inference census - RefreshLatestSignal was NEVER CALLED (0 attempts). The new-bar gate never fired.");
return;
}
Print(ID + ": inference census - ", attempts, " refresh attempts: ", m_refreshOk, " completed, ",
m_refreshFailFeatures, " bailed in BufferTempData, ", m_refreshFailShort, " bailed on a short feature window",
" | decisions Buy:", m_refreshBuy, " Sell:", m_refreshSell, " Neutral:", m_refreshNeutral);
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 20:39:49 -04:00
//--- Second half of the census, and the half that separates "the model said nothing" from "the model
//--- spoke and was not allowed to vote" - see m_voteGateBlocked for why that distinction is the whole
//--- point. A run with directional decisions above and voteGate passed:0 below is the readiness-gate
//--- failure, NOT a Neutral model, and the fix is on the model-load path.
if(m_voteGateCompleteAtFirst < 0)
Print(ID + ": inference census - vote gate was NEVER REACHED (no directional decision ever hit "
"LongCondition/ShortCondition). Either every decision was Neutral, or this filter was never polled.");
else
Print(ID + ": inference census - vote gate passed:", m_voteGatePassed, " blocked:", m_voteGateBlocked,
" | at first vote trainingComplete=", (m_voteGateCompleteAtFirst != 0 ? "true" : "false"),
" modelLoadedFromDisk=", (m_voteGateLoadedAtFirst != 0 ? "true" : "false"),
" inferenceOnly=", (m_inferenceOnly ? "true" : "false"),
(m_voteGateBlocked > 0 && m_voteGatePassed == 0
? " <-- EVERY directional call was discarded here. This is the zero-direction cause."
: ""));
}
void StartTraining(void)
{
if(!m_trainingStopRequested && !m_trainingPaused)
return;
m_trainingStopRequested = false;
m_trainingPaused = false;
if(!bEventStudy)
bEventStudy = EventChartCustom(ChartID(), 1, (long)dtStudied, 0, "Resume");
Print(ID + ": training (re)started by user (era " + IntegerToString(m_eraCount) + ")");
}
//--- Has an era ever cleared the per-class recall floor and been checkpointed this run? This is the
//--- same quality bar the plateau ladder's auto-deploy requires (see PLATEAU_STAGE_DEPLOY), exposed so
//--- the panel can warn before a MANUAL deploy ships a model that ignores Buy or Sell.
bool HasRecallPassingCheckpoint(void) const { return m_bestPassedRecall; }
//--- MANUAL deploy (panel "Deploy Model"): finalise whatever the run has found so far as THE model -
//--- exactly what the plateau ladder does on its own at stage 3, just triggered early by the operator.
//--- Restores the best checkpoint (not whatever era the run happened to be mid-way through), persists
//--- weights + calibration + shadow, and flips to live inference.
//--- Deliberately does NOT set m_trainingStopRequested: like every other deploy path, a deployed model
//--- still runs live inference AND online continual learning. A panel Stop is the thing that halts
//--- everything - see StopTraining()/OnlineLearnStep()'s gate. Reversible via RetrainDeployed().
bool DeployNow(void)
{
if(CheckPointer(Net) == POINTER_INVALID || !m_isInitialized)
return false;
if(m_trainingComplete)
return true; // already deployed - nothing to do
//--- Set BEFORE any save below: the flag is written INTO the .nnw, so persisting first would store
//--- "still training" and a restart would resume the era loop instead of running the deployed model.
m_trainingComplete = true;
m_trainingPaused = false;
m_trainingStopRequested = false;
if(m_trainRunActive || m_haveOosCheckpoint)
FinalizeTrainRun(); // restores the best checkpoint, persists, ends the run
else
{
//--- Nothing trained this session (e.g. deploying a model that was just loaded from disk), so
//--- there is no in-memory checkpoint to restore - persist exactly what is loaded right now.
PersistDeployedModel();
SaveChartSignals();
}
RefreshLatestSignal();
Print(ID + ": model DEPLOYED by user at era " + IntegerToString(m_eraCount) +
" (balanced accuracy " + (m_bestBalancedOos < 0 ? "n/a" : DoubleToString(m_bestBalancedOos, 1) + "%") +
", blended OOS " + DoubleToString(dOosForecast, 1) + "%) - training stopped, now running live inference" +
(m_enableOnlineLearning ? " with online continual learning" : "") +
". Use the panel's \"Retrain Model\" to resume training from here.");
feat: gate deployment on the null of the MAXIMUM, not the per-era null EDGE_MIN_SIGMAS is a PER-ERA test and the deployed model is the MAXIMUM over every era a run ranks. A 2-sigma one-sided test passes on noise with probability 0.0228 per era, so over N eras the chance at least one clears it is 1-(1-0.0228)^N: 34% by era 18, 80% by era 70, 93% by era 112. The gate was near-certain to open on a long run whatever the data held. It did. HYBRID deployed 2026-08-08 at dir-precision 35.5% vs 34% chance - +1.5pp, best of 112 eras whose per-era values wandered 30%..35.5%. At the call counts these runs produce that is p_family 0.92..0.9999. Every OTHER best-of-N decision here already carries this correction, and every one REJECTS on this data: the barrier-geometry winner (null of the maximum over 6, p=0.3902), the indicator tuner (Sidak, p=1.0000), the MI lag profile (null of the maximum over 21 lags). The one decision that ships a model to a live account had none. BestCheckpointSurvivesSelection() re-tests the checkpoint that is about to deploy: z = (precision - chance)/SE, SE = sqrt(p0(1-p0)/n) p_single = P(Z >= z) p_family = 1 - (1-p_single)^N against DEPLOY_FAMILY_WISE_ALPHA. It uses the checkpoint's OWN snapshotted precision/chance/call-count, not the latest era's, because the model that ships is the one that has to clear the bar. N counts CANDIDATE eras (coverage measurable, at least one directional call) - an era that called nothing directional could never have become the best, so counting it would make the gate stricter than the search that actually happened. Conservative on purpose: consecutive eras share OOS bars and differ by one gradient step, so they are nowhere near N independent draws and the true family-wise error is below this bound. This gate decides what trades real money and the house posture is reject-unless-demonstrated. Effect at 2900 directional calls / N=112: required edge goes 1.76pp -> 2.92pp. A real edge clears it; +1.5pp does not. Applied to BOTH automatic paths - the plateau ladder's stage-3 deploy and the m_trainingComplete assignment - which must stay identical or the flag persisted into the .nnw disagrees with the decision to stop, and a reload runs inference on a model the ladder refused. NOT applied to the two operator paths (era-cap deploy, panel Deploy button). Those stay the operator's call; ReportSelectionGateVerdict() logs the verdict beside them so an authorised deploy can never later be misread as a validated one. NormalUpperTail() is A&S 26.2.17 (|err| < 7.5e-8), self-contained rather than pulling in Math\Stat. Verified against reference values to 6dp: Q(1.645)=0.049985, Q(1.96)=0.024998, Q(3.0)=0.001350. Its locals are ntB1..ntB5 because AI\Network.mqh line 79 does "#define b1 AdamBeta1". Compiles clean: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-08 18:01:04 -04:00
//--- Deliberately reported, not enforced: a manual deploy is the operator overriding the ladder, and
//--- that override stays available. But the ladder's own verdict on this checkpoint goes in the log
//--- next to it, so "I shipped this" is never later mistaken for "this passed". See
//--- DEPLOY_FAMILY_WISE_ALPHA and HasRecallPassingCheckpoint()'s panel warning.
ReportSelectionGateVerdict("manual deploy");
return true;
}
//--- The inverse of DeployNow(), and the ONLY way back: while m_trainingComplete is set,
//--- ScheduleTrainingIfNeeded() routes every tick to the converged/inference branch, so StartTraining()
//--- alone can never revive a deployed model (it clears the stop flag, but the complete flag still wins
//--- that branch). Clearing it here re-arms the normal training path, continuing from the DEPLOYED
//--- weights rather than from scratch - "reset weights" is the separate, destructive button for that.
//--- The plateau ladder, best-checkpoint tracking and annealed gamma all reset themselves when Train()
//--- starts its next fresh run (see the !m_trainRunActive block), so a retrain does not inherit the
//--- exhausted stage that deployed the model and immediately re-deploy it.
void RetrainDeployed(void)
{
if(!m_trainingComplete)
return;
m_trainingComplete = false;
m_trainingStopRequested = false;
m_trainingPaused = false;
//--- Persist the cleared flag immediately. Otherwise a terminal restart before the first era
//--- completes would reload the .nnw still marked complete and silently go back to inference-only,
//--- looking like the button did nothing.
PersistDeployedModel();
if(!bEventStudy)
bEventStudy = EventChartCustom(ChartID(), 1, (long)dtStudied, 0, "Retrain");
Print(ID + ": RETRAINING the deployed model from era " + IntegerToString(m_eraCount) +
" - keeping its current weights as the starting point (use \"Delete & Reset Weights\" to start from scratch instead).");
}
//--- Manual "rescan" of the drawn signal arrows: purges every arrow currently on the chart (namespaced
//--- delete - user drawings untouched) and re-infers the last SIGNAL_RESCAN_LOOKBACK_BARS bars from the
//--- CURRENTLY deployed weights, then re-runs the same end-of-era NMS declutter (PruneDirectionalClusters)
//--- used during training so the fresh set matches what a live re-render would have produced. Wired to
//--- the panel's Hide->Show Signals sequence: without this, "restore" only ever replays whatever was
//--- last saved to the .arrows sidecar, which for a long-deployed model can be a stale historical render
//--- from whenever it was last actually trained - years-old arrows crowding out anything recent. Chart-only
//--- (no persistent chart in the tester/optimizer) and a no-op until a model has something to infer with.
//--- This only does the cheap setup (buffer resize, arrow purge, cache alloc) and QUEUES the per-bar
//--- inference loop for AdvanceChartSignalRescan() to drain in time-boxed slices off the timer - see
//--- that method's comment for why the loop itself must never run in one blocking pass. Returns true
//--- once a rescan has been queued (check RescanPending() for completion), false if there was nothing
//--- to rescan (no deployed model, tester/optimizer context, etc).
bool StartChartSignalRescan(void)
{
if(MQLInfoInteger(MQL_TESTER) || MQLInfoInteger(MQL_OPTIMIZATION) || MQLInfoInteger(MQL_FORWARD))
return false;
if(CheckPointer(Net) == POINTER_INVALID || !m_isInitialized || !m_trainingComplete)
return false;
if(m_outputNeuronsCount != 1 && m_outputNeuronsCount != 3)
return false;
int barsAvail = Bars(m_symbol.Name(), PERIOD_CURRENT);
int barsNow = MathMin(SIGNAL_RESCAN_LOOKBACK_BARS, barsAvail);
if(barsNow <= m_historyBars)
return false;
if(!ResizeBuffers(barsNow) || !RefreshData())
return false;
EnsureShadowNet();
//--- Drop only the arrows THIS rescan is about to re-judge - i.e. those within [now, oldest bar of
//--- the barsNow window] - not every namespaced arrow on the chart. PruneDirectionalClusters below
//--- only touches indices inside [0, barsNow), so anything older survives untouched either way; the
//--- previous version called ObjectsDeleteAll(0, SIG_ARROW_PREFIX) unconditionally, which wiped
//--- EVERY arrow ever drawn (years of history, going back to whenever this EA was first attached)
//--- and then only ever redrew the last SIGNAL_RESCAN_LOOKBACK_BARS (~7 months on H1) - anything
//--- older was gone permanently the moment Show Signals was clicked, with no way back short of the
//--- .arrows sidecar (itself only ever a snapshot from whatever the LAST save happened to catch).
//--- Reported 2026-07-26: a user's multi-year arrow history vanished down to whatever a stale
//--- .arrows file held, immediately after their first-ever successful Show Signals click. This scoped
//--- delete is the fix - older arrows are never in scope to be wiped in the first place.
datetime rescanCutoffTime = m_Time.GetData(barsNow - 1);
for(int oi = ObjectsTotal(0, 0, OBJ_ARROW) - 1; oi >= 0; oi--)
{
string onm = ObjectName(0, oi, 0, OBJ_ARROW);
if(StringFind(onm, SIG_ARROW_PREFIX) != 0)
continue;
if((datetime)ObjectGetInteger(0, onm, OBJPROP_TIME) >= rescanCutoffTime)
ObjectDelete(0, onm);
}
ArrayResize(m_arrowSignalCache, barsNow);
ArrayInitialize(m_arrowSignalCache, -2.0);
m_rescanBarsNow = barsNow;
m_rescanHi = barsNow - m_historyBars;
m_rescanIndex = 0;
m_rescanRawBuy = 0;
m_rescanRawSell = 0;
m_rescanRawNeutral = 0;
m_rescanPending = (m_rescanHi > 0);
m_rescanStartMs = GetTickCount();
if(m_rescanPending)
Print(ID + ": rescanning last " + IntegerToString(m_rescanHi) + " bars against the deployed model (progressive, non-blocking)...");
return m_rescanPending;
}
//--- true while a queued rescan (StartChartSignalRescan above) still has slices left for
//--- AdvanceChartSignalRescan to drain - polled by Warrior_EA.mq5's FinalizeSignalsRescanIfDone() to
//--- know when it's safe to (re)apply arrow visibility and report the Show Signals click as complete.
bool RescanPending(void) const { return m_rescanPending; }
//--- forces a save of the network's current in-memory weights/state regardless of era-completion
//--- state; called from OnDeinit() so shutdown/chart-removal never loses more than the current tick
//--- of learning, and a subsequent restart's Train() resumes from m_eraCount rather than the last
//--- fully-completed era only.
//--- Heavy weight/state persistence ONLY (net weights + calibration sidecar + shadow net). Split out
//--- from PersistOnShutdown() so OnDeinit() can run the cheap chart save+purge FIRST: on the CPU-DLL
//--- box this recursive save (two full nets) is the slow/fragile step, and if it ever stalls past MT5's
//--- deinit budget or faults, the arrows and status panel must already be gone, not stranded on the
//--- chart (the reported "cleanup not going well - signals/panel stay after Abnormal termination").
bool PersistWeightsOnShutdown(void)
{
if(CheckPointer(Net) == POINTER_INVALID || !m_isInitialized)
return false;
//--- An inference-only run (any Strategy Tester pass - see m_inferenceOnly) trains NOTHING, so there
//--- is no new state to persist and this save can only do harm. It is exactly what corrupted the
//--- tester cache on 2026-07-26: when the seeded load failed, the !netLoaded path reset the in-memory
//--- state to a fresh untrained net (era 0), and this unconditional shutdown save then wrote THAT
//--- over the good seeded copy - which, because seeding only re-runs when the cache file is absent,
//--- silently poisoned every subsequent backtest. Skipping it makes the tester cache strictly
//--- read-only for single backtests: it can only ever be (re)written by the seed copy from
//--- production, never by a run's own in-memory state.
if(m_inferenceOnly)
{
PrintVerbose(ID + ": inference-only run - skipping the shutdown weight save (nothing was trained; the cached model is left exactly as seeded).");
return true;
}
fix(ai): stop the shutdown save from resurrecting reset weights; size HYBRID's LSTM to its real fan-in ResetWeights already deletes the whole model set - .nnw, .cfg, _ckpt.tmp, .stats, _shadow.nnw - and clears both the .arrows sidecar and the drawn chart objects. What undid it was PersistWeightsOnShutdown: detaching the EA after a reset but before an era completed re-created a .nnw from the freshly-built, never-run net, so the next attach loaded an era-0 stub instead of starting clean. For LSTM/HYBRID that stub is worse than nothing - a layer that has never run a forward pass has m_iInputs<=0, so Save omits every LSTM buffer (see 413ff7e). Skip the save when no era completed and no model was loaded; that is exactly the post-reset and first-attach state. Also sweep _shadowclone.tmp, which the reset did not cover. Separately, ComputeLstmHiddenSize budgeted every topology against the flattened input (historyBars x neuronsCount). True for LSTM, wrong for HYBRID, where AddConvStage runs first and the LSTM is fed the conv feature map - historyBars x convFilterCount, 160 rather than 420 at H1 defaults. The quadratic is dominated by the inputs term, so overstating the fan-in 2.6x cost a full ladder step (16 units where the budget affords 32). New virtual HasConvBeforeLstm() feeds LstmFanIn(), so composition decides this rather than an AIType check. desc.window is advisory only - CNet never passes it to the layer - but is now truthful for the same reason. Derived values stay out of the weights-filename fingerprint and are adopted from the .cfg, so existing models keep their saved width; only fresh ones pick up the corrected budget. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 13:06:09 -04:00
//--- Nothing trained and nothing loaded => there is no state to persist, and writing anyway is
//--- actively harmful. This is what defeated the panel's reset-weights button: ResetWeights
//--- correctly deletes the .nnw/.cfg/.stats/_ckpt/_shadow set and the .arrows sidecar, but if the
//--- EA is then detached before a single era completes, THIS save immediately re-created a .nnw
//--- from the freshly-built, never-run net - so the next attach loaded an era-0 stub instead of
//--- starting clean. For LSTM/HYBRID that stub is worse than useless: a layer that has never run a
//--- forward pass has m_iInputs<=0, so CNeuronLSTMOCL::Save omits every LSTM buffer (see the note
//--- in its Load()). m_eraCount==0 && !m_modelLoadedFromDisk is exactly the state ResetWeights
//--- leaves behind, and also the first-ever-attach state - both cases have nothing worth writing.
if(m_eraCount == 0 && !m_modelLoadedFromDisk)
{
PrintVerbose(ID + ": no era completed and no model loaded - skipping the shutdown weight save (leaving the model files absent so the next attach starts genuinely clean).");
return true;
}
double currentIndicatorParams[];
m_indicatorTuner.Flatten(currentIndicatorParams);
bool ok = Net.Save(m_activeFileName + ".nnw", dError, dUndefine, dForecast, dtStudied, m_activeFileCommon, m_eraCount, m_trainingComplete, currentIndicatorParams);
//--- calibration state (class priors + confidence scale) must travel with the weights so live
//--- trading behaves like training after a restart - see SaveModelStats().
if(!SaveModelStats(m_activeFileName, m_activeFileCommon))
Print(ID + ": ERROR - shutdown SaveModelStats failed for " + m_activeFileName + ". Calibration state not persisted.");
//--- Deliberately do NOT save the shadow net here. On the CPU-DLL box a second full-net write
//--- (~18MB) roughly DOUBLES the shutdown cost, and OnDeinit has a limited budget before MT5 reports
//--- "Abnormal termination" and skips the rest of the teardown. (An earlier revision also blamed that
//--- overrun for the "failed to allocate layer 0" reloads; that was a misdiagnosis - the real cause
//--- was a lost virtual override in the read path, see AI\Network.mqh's CLayer::CreateElement note.
//--- Halving the shutdown cost is still worth it on its own.) The shadow is NOT lost: it is saved
//--- every era (Train()), at FinalizeTrainRun(), and every ONLINE_LEARN_PERSIST_EVERY bars during
//--- online learning, and it self-heals (EnsureShadowNet re-blends from the main Net when
//--- missing/stale). Worst case a
//--- shutdown loses only the shadow's in-progress-era drift, which re-converges - a far better
//--- trade than risking the whole model to an over-budget shutdown.
if(!ok)
Print(ID + ": ERROR - failed to persist weights on shutdown for " + m_activeFileName + ", error " + IntegerToString(GetLastError()));
else
PrintVerbose(ID + ": weights persisted on shutdown (era " + IntegerToString(m_eraCount) + ", trainingComplete=" + (string)m_trainingComplete + ")");
return ok;
}
//--- Persist the drawn arrows to disk, then remove THIS EA's chart visuals (arrows + status label).
//--- Called early in OnDeinit(), before the heavy weight save, so a later stall/fault in the save can
//--- never leave the chart littered. Idempotent - the destructor's PurgeChart() then simply no-ops.
//--- Deliberately NOT part of SaveWeightsNow(): a mid-session manual save must not wipe the chart.
fix(chart): arrows survived the EA that drew them - persist, then clear Reported: on deinit the panel and status label go, the signal arrows stay. Two independent causes, both fixed here. 1. It was partly deliberate. ShutdownChartCleanup carried a second behaviour selected by a `preserveChartArrows` flag derived from the deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the arrows were left on the chart on purpose, to avoid a reload flicker. That branch IS the reported symptom, an operator cannot tell it apart from a cleanup that failed, and it was outright wrong whenever the reload changed the config - REASON_PARAMETERS means exactly that, and the preserved arrows then belonged to a model the chart no longer runs, with nothing marking them stale. It is gone, along with the flag and m_purgeChartOnDestruct. One path now: persist, clear, restore on the next attach. 2. Whatever remains was unfalsifiable. PurgeChart was a single ObjectsDeleteAll(prefix) whose return value was discarded, with no caller ever looking at the chart again - so "the arrows are still there" and "the arrows were never there" produced identical evidence, which is why the report survived three sessions. It now verifies: after the bulk delete it walks the OBJ_ARROW-typed list (a handful of objects, not the whole chart), deletes any surviving WarSig_ by name, and says so. Costs one typed scan when the bulk delete works, which is the normal case; names the root cause when it does not. Every failure mode of SaveChartSignals was also silent - it returned void and had three bare early returns. It returns bool now, logs the open error with the filename, and the shutdown purge is CONDITIONAL on it: for a converged model the chart objects are the only copy of its signal history (nothing redraws them - the renderer runs per training era and a deployed model has none left), so a chart left littered because the disk write failed beats a clean chart bought by destroying the history. Either way the log now says which happened. Also states the user's rule once, where arrows come back rather than across InitNeuralNetwork's several exits: no weights loaded for this config => clear the sidecar and start visually clean. A fresh run must not inherit calls it never made, and the first save would otherwise adopt them (the sidecar is rebuilt by scanning the chart). Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
//---
//--- 2026-08-01: there used to be a SECOND behaviour here, selected by a `preserveChartArrows` flag the
//--- caller derived from the deinit reason - on a recompile / input change / template swap / symbol
//--- change the arrows were deliberately LEFT on the chart, on the theory that an in-process reload
//--- should not flicker. It is gone, and the flag with it. Three reasons, in order of weight:
//--- 1. It is the reported defect. "The EA removes its panel and label but its signals stay" is what
//--- that branch does by design, and an operator has no way to tell a deliberate warm-reload
//--- preserve from a cleanup that failed.
//--- 2. It was only correct when the reload keeps the SAME config. Change an input that feeds the
//--- weights fingerprint - which is exactly what REASON_PARAMETERS means - and the preserved
//--- arrows belong to a model this chart is no longer running, with nothing to mark them stale.
//--- 3. The save/restore mechanism it was avoiding already handles this case, progressively and
//--- without blocking OnInit (LoadChartSignals + AdvanceChartSignalRestore). Keeping a second,
//--- subtly different route to the same outcome bought a few hundred milliseconds of flicker.
//--- One path now: persist, clear, restore on the next attach if a model for this config exists.
void ShutdownChartCleanup(void)
{
fix(chart): arrows survived the EA that drew them - persist, then clear Reported: on deinit the panel and status label go, the signal arrows stay. Two independent causes, both fixed here. 1. It was partly deliberate. ShutdownChartCleanup carried a second behaviour selected by a `preserveChartArrows` flag derived from the deinit reason: on RECOMPILE / PARAMETERS / CHARTCHANGE / TEMPLATE the arrows were left on the chart on purpose, to avoid a reload flicker. That branch IS the reported symptom, an operator cannot tell it apart from a cleanup that failed, and it was outright wrong whenever the reload changed the config - REASON_PARAMETERS means exactly that, and the preserved arrows then belonged to a model the chart no longer runs, with nothing marking them stale. It is gone, along with the flag and m_purgeChartOnDestruct. One path now: persist, clear, restore on the next attach. 2. Whatever remains was unfalsifiable. PurgeChart was a single ObjectsDeleteAll(prefix) whose return value was discarded, with no caller ever looking at the chart again - so "the arrows are still there" and "the arrows were never there" produced identical evidence, which is why the report survived three sessions. It now verifies: after the bulk delete it walks the OBJ_ARROW-typed list (a handful of objects, not the whole chart), deletes any surviving WarSig_ by name, and says so. Costs one typed scan when the bulk delete works, which is the normal case; names the root cause when it does not. Every failure mode of SaveChartSignals was also silent - it returned void and had three bare early returns. It returns bool now, logs the open error with the filename, and the shutdown purge is CONDITIONAL on it: for a converged model the chart objects are the only copy of its signal history (nothing redraws them - the renderer runs per training era and a deployed model has none left), so a chart left littered because the disk write failed beats a clean chart bought by destroying the history. Either way the log now says which happened. Also states the user's rule once, where arrows come back rather than across InitNeuralNetwork's several exits: no weights loaded for this config => clear the sidecar and start visually clean. A fresh run must not inherit calls it never made, and the first save would otherwise adopt them (the sidecar is rebuilt by scanning the chart). Compiles 0 errors / 0 warnings, standard and Market. Needs redeploy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 13:28:34 -04:00
PersistAndClearChartSignals();
}
//--- Full shutdown persistence (weights + arrows), preserved for the panel's manual "save weights"
//--- button (SaveWeightsNow) - does NOT purge the chart. OnDeinit no longer calls this; it runs
//--- ShutdownChartCleanup() then PersistWeightsOnShutdown() so cleanup can't be starved by the save.
bool PersistOnShutdown(void)
{
bool ok = PersistWeightsOnShutdown();
//--- Persist the drawn arrows too so a re-add/recompile restores them without a retrain.
SaveChartSignals();
return ok;
}
//--- explicit manual save, identical persistence to PersistOnShutdown() but user-triggered from the panel
bool SaveWeightsNow(void) { return PersistOnShutdown(); }
//--- reloads this signal's current-config weights file from disk, discarding any unsaved in-memory
//--- training progress since the last successful save
bool LoadWeightsNow(void)
{
if(CheckPointer(Net) == POINTER_INVALID)
return false;
double loadedIndicatorParams[];
bool netLoaded = Net.Load(m_activeFileName + ".nnw", dError, dUndefine, dForecast, dtStudied, m_activeFileCommon, m_eraCount, m_trainingComplete, loadedIndicatorParams);
if(!netLoaded)
{
Print(ID + ": ERROR - failed to load weights from " + m_activeFileName + ".nnw, error " + IntegerToString(GetLastError()));
return false;
}
m_modelLoadedFromDisk = true;
fix: stop a .nnw from pinning a superseded architecture A .nnw persists the ARCHITECTURE, not just the weights: Save writes (int)activation per neuron and Load reads it straight back. The activation chosen in BuildFreshTopology() therefore only ever reached a brand-new topology - every reload restored the file's value and the next save wrote it back out, so a wrong value could never heal while the source read as though it were already fixed. That is how five models kept training with an unbounded NONE classification head for a full day after the 07-28 revert to SIGMOID. Confirmed by parsing the binaries: 848cb42c.nnw / 2e754b43.nnw carry `act=NONE` on the 3-neuron output layer, while a genuinely reset model of the same config carries act=SIGMOID. In the log it showed as negative "OOS raw out" values - impossible under sigmoid - escalating to a 4.14e13 logit spread with all three classes numerically identical (input-independent output) and balanced accuracy pinned on the 33.3% one-class floor. - OutputLayerActivation() is now the single source of truth, called by both BuildFreshTopology() and the new load-time repair, so the two can no longer diverge the way a duplicated literal did. - CNet::EnforceOutputActivation() re-asserts it after Load and reports the stale value; CExpertSignalAIBase::EnforceTopologyContract() logs the repair loudly, since weights learned under the old head may not be worth keeping even once the head is corrected. - Hidden layers are deliberately left alone: they legitimately differ per stage (PRELU dense/conv, NONE pool, TANH LSTM). Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 12:00:40 -04:00
//--- the file may carry a superseded architecture - correct it before anything reads the net
EnforceTopologyContract();
//--- restore the calibration state that pairs with these weights (priors + confidence scale) so a
//--- manual reload keeps live decisions calibrated exactly as the saved model was - see LoadModelStats().
LoadModelStats(m_activeFileName, m_activeFileCommon);
if(ArraySize(loadedIndicatorParams) == AD_TUNE_PARAM_COUNT)
{
2026-08-13 10:23:11 -04:00
//--- same no-change guard as the resume path - see AdoptIndicatorParams
if(m_indicatorsPtr != NULL)
2026-08-13 10:23:11 -04:00
AdoptIndicatorParams(loadedIndicatorParams, m_indicatorsPtr);
else
m_indicatorTuner.Unflatten(loadedIndicatorParams);
}
//--- this just swapped dtStudied/m_eraCount/Net's weights out from under whatever a chunked
//--- Train() run (see its declaration comment) had cached for the era/trial it was mid-way
//--- through - discard that resumable state so the next Train() call starts a fresh era
//--- against the just-loaded dtStudied instead of resuming bar-loop bookkeeping computed
//--- against a now-stale one.
m_trainRunActive = false;
m_eraResumePending = false;
m_haveOosCheckpoint = false;
m_oosWindow.Clear();
m_tuneTrialIndex = -1;
RefreshLatestSignal();
Print(ID + ": weights reloaded from disk (era " + IntegerToString(m_eraCount) + ", trainingComplete=" + (string)m_trainingComplete + ")");
return true;
}
//--- deletes this signal's current-config saved files only (weights, topology config, in-progress
//--- checkpoint) and rebuilds a fresh untrained topology in memory so training restarts from era 0.
//--- Never touches another signal type's or another symbol/timeframe's files - m_fileName already
//--- embeds symbol+period+id+output-count+opt-algo.
bool ResetWeights(void)
{
bool stopped = m_trainingStopRequested;
m_trainingStopRequested = true; // hold off any in-flight Train() scheduling while we reset
//--- targets whichever file this run is actually training against (see InitNeuralNetwork): the
//--- shared production weights normally, or the local tester/optimizer cache during a backtest -
//--- so resetting from the panel during a visual-mode backtest can never wipe the live model.
int flags = m_activeFileCommon ? FILE_COMMON : 0;
string nnw = m_activeFileName + ".nnw";
string cfg = m_activeFileName + ".cfg";
string ckpt = m_activeFileName + "_ckpt.tmp";
//--- The calibration/online-learning sidecar (.stats: priors, confidence scale, CPU-inference marker,
//--- and the online watermark/guardrail - see SaveModelStats) and the deployed EMA shadow
//--- (_shadow.nnw - see SaveShadowNet) both pair with the weights being erased. Delete them too, or
//--- a fresh retrain would silently inherit the OLD model's calibration and blend into a stale shadow
//--- (EnsureShadowNet loads _shadow.nnw from disk before it ever clones the new Net).
string stats = m_activeFileName + ".stats";
string shadow = m_activeFileName + "_shadow.nnw";
fix(ai): stop the shutdown save from resurrecting reset weights; size HYBRID's LSTM to its real fan-in ResetWeights already deletes the whole model set - .nnw, .cfg, _ckpt.tmp, .stats, _shadow.nnw - and clears both the .arrows sidecar and the drawn chart objects. What undid it was PersistWeightsOnShutdown: detaching the EA after a reset but before an era completed re-created a .nnw from the freshly-built, never-run net, so the next attach loaded an era-0 stub instead of starting clean. For LSTM/HYBRID that stub is worse than nothing - a layer that has never run a forward pass has m_iInputs<=0, so Save omits every LSTM buffer (see 413ff7e). Skip the save when no era completed and no model was loaded; that is exactly the post-reset and first-attach state. Also sweep _shadowclone.tmp, which the reset did not cover. Separately, ComputeLstmHiddenSize budgeted every topology against the flattened input (historyBars x neuronsCount). True for LSTM, wrong for HYBRID, where AddConvStage runs first and the LSTM is fed the conv feature map - historyBars x convFilterCount, 160 rather than 420 at H1 defaults. The quadratic is dominated by the inputs term, so overstating the fan-in 2.6x cost a full ladder step (16 units where the budget affords 32). New virtual HasConvBeforeLstm() feeds LstmFanIn(), so composition decides this rather than an AIType check. desc.window is advisory only - CNet never passes it to the layer - but is now truthful for the same reason. Derived values stay out of the weights-filename fingerprint and are adopted from the .cfg, so existing models keep their saved width; only fresh ones pick up the corrected budget. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 13:06:09 -04:00
//--- EnsureShadowNet clones the live net through this temp file (see CloneNetInto). A crash or a
//--- reset mid-clone leaves it on disk shaped for the model being erased; sweep it with the rest.
string shadowClone = m_activeFileName + "_shadowclone.tmp";
ResetLastError();
if(FileIsExist(nnw, flags) && !FileDelete(nnw, flags))
Print(ID + ": ERROR - failed to delete " + nnw + ", error " + IntegerToString(GetLastError()));
ResetLastError();
if(FileIsExist(cfg, flags) && !FileDelete(cfg, flags))
Print(ID + ": ERROR - failed to delete " + cfg + ", error " + IntegerToString(GetLastError()));
ResetLastError();
if(FileIsExist(ckpt, flags) && !FileDelete(ckpt, flags))
Print(ID + ": ERROR - failed to delete " + ckpt + ", error " + IntegerToString(GetLastError()));
ResetLastError();
if(FileIsExist(stats, flags) && !FileDelete(stats, flags))
Print(ID + ": ERROR - failed to delete " + stats + ", error " + IntegerToString(GetLastError()));
ResetLastError();
if(FileIsExist(shadow, flags) && !FileDelete(shadow, flags))
Print(ID + ": ERROR - failed to delete " + shadow + ", error " + IntegerToString(GetLastError()));
fix(ai): stop the shutdown save from resurrecting reset weights; size HYBRID's LSTM to its real fan-in ResetWeights already deletes the whole model set - .nnw, .cfg, _ckpt.tmp, .stats, _shadow.nnw - and clears both the .arrows sidecar and the drawn chart objects. What undid it was PersistWeightsOnShutdown: detaching the EA after a reset but before an era completed re-created a .nnw from the freshly-built, never-run net, so the next attach loaded an era-0 stub instead of starting clean. For LSTM/HYBRID that stub is worse than nothing - a layer that has never run a forward pass has m_iInputs<=0, so Save omits every LSTM buffer (see 413ff7e). Skip the save when no era completed and no model was loaded; that is exactly the post-reset and first-attach state. Also sweep _shadowclone.tmp, which the reset did not cover. Separately, ComputeLstmHiddenSize budgeted every topology against the flattened input (historyBars x neuronsCount). True for LSTM, wrong for HYBRID, where AddConvStage runs first and the LSTM is fed the conv feature map - historyBars x convFilterCount, 160 rather than 420 at H1 defaults. The quadratic is dominated by the inputs term, so overstating the fan-in 2.6x cost a full ladder step (16 units where the budget affords 32). New virtual HasConvBeforeLstm() feeds LstmFanIn(), so composition decides this rather than an AIType check. desc.window is advisory only - CNet never passes it to the layer - but is now truthful for the same reason. Derived values stay out of the weights-filename fingerprint and are adopted from the .cfg, so existing models keep their saved width; only fresh ones pick up the corrected budget. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 13:06:09 -04:00
ResetLastError();
if(FileIsExist(shadowClone, flags) && !FileDelete(shadowClone, flags))
Print(ID + ": ERROR - failed to delete " + shadowClone + ", error " + IntegerToString(GetLastError()));
fix: clear stale signal arrows when a fresh model starts at era 0 Arrow cleanup existed on two paths - the panel's reset-weights, and the topology-mismatch discard - but both are gated on there being a saved .nnw to delete. The third case had no cleanup at all: a fresh topology at era 0 with no weights behind it, which is what a changed config produces. A new fingerprint makes a new m_fileName, so the previous model's files are not "discarded", they are simply not this model's files, and nothing ever cleared the chart. That is not cosmetic. Arrows outlive the model that drew them twice over: 1. The chart objects live in the CHART, not the sidecar, so they survive a remove/re-add, a recompile, a restart and a fresh deploy no matter what happens to any file on disk. 2. SaveChartSignals() rebuilds the sidecar by SCANNING the chart for SIG_ARROW_PREFIX objects. So the first save of the fresh run adopts the dead model's calls and writes them out under the NEW model's filename - laundering them into the new model's history where nothing can separate them afterwards. Extracted the duplicated cleanup into ClearPersistedChartSignals(reason) - it cancels the deferred restore queue, deletes m_fileName + ".arrows", clears the namespaced chart objects and logs why - and called it from all three paths. The call sits at the BuildFreshTopology() call site, not inside it: the genetic tuner rebuilds a throwaway topology per candidate (AutoTune.mqh) and must never touch the chart. All three sites run after m_fileName has its config fingerprint appended, so they target the right sidecar. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 10:13:01 -04:00
//--- The drawn signal arrows and their .arrows sidecar belong to the model being erased, exactly
//--- like the .stats/_shadow sidecars above - see ClearPersistedChartSignals().
ClearPersistedChartSignals("weights reset from the panel");
m_eraCount = 0;
m_trainingComplete = false;
m_modelLoadedFromDisk = false;
dtStudied = 0;
dError = -1;
dUndefine = 0;
dForecast = 0;
dPrevSignal = 0;
m_nmsLiveBuyTime = 0;
m_nmsLiveSellTime = 0;
m_nmsLiveBuyAccept = false;
m_nmsLiveSellAccept = false;
m_nmsLiveKeptTime = 0;
m_nmsLiveKeptDir = Neutral;
m_nmsLiveKeptConf = 0;
dOosError = -1;
dOosForecast = 0;
m_oosSamples = 0;
//--- fresh model => wipe the compounded/persistent accuracy history too (it is only reset here; a
//--- normal restart restores it from .stats, and it survives era-to-era). See m_cumIsCorrect.
m_cumIsCorrect = 0;
m_cumIsTotal = 0;
m_cumOosCorrect = 0;
m_cumOosTotal = 0;
//--- discard any in-progress chunked run/tuning state - it references buffers/checkpoints from
//--- before this reset and must never be resumed into the freshly rebuilt topology below
m_trainRunActive = false;
m_eraResumePending = false;
m_haveOosCheckpoint = false;
m_oosWindow.Clear();
m_syncWaitStartTick = 0;
m_tuneTrialIndex = -1;
//--- an explicit reset restarts from era 0 against a fresh topology - re-verify history sync and
//--- rebuild the label cache too, and drop any in-flight continual-learning OOS simulation, since
//--- both would otherwise reference bars/weights from before this reset.
m_warmupPassesRemaining = 3;
m_labelCacheBars = 0;
m_labelCacheAnchorTime = 0;
m_labelCachePrebuilt = false;
m_labelPrebuildActive = false;
m_prebuildSeedPending = false;
if(m_simOosRunActive)
{
delete m_simOosNet;
m_simOosNet = NULL;
m_simOosRunActive = false;
}
//--- Re-seed before building a fresh topology so weight init is genuinely random, not dominated
//--- by whatever fixed/deterministic seed the genetic tuner's last candidate evaluation left in
perf(autotune): replace the genetic search with a filter score - hours to seconds MEASURED COST OF THE GA, which is what retired it. Per generation: rung 0: 8 cand x 3 seeds x 3 eras = 72 eras rung 1: 4 cand x 3 seeds x 8 eras = 96 rung 2: 2 cand x 3 seeds x 20 eras = 120 = 288 eras/generation x 4 generations = 1152 eras BEFORE the winner's real training began. Against the observed era times on SP500 H1: PAI 29.1 s/era -> 9.3 h (matches the observed 00:37 -> 09:22) CONV 41.3 s/era -> 13.2 h LSTM 150.4 s/era -> 48.1 h HYBRID 154.6 s/era -> 49.5 h Two days to tune is not a first-run experience, and it is the phase in which the panel goes quiet, which is what made it look like a hang. It also bought nothing. The space is 90 points (10 MA periods x 9 MA types), so 1152 evaluations revisited each point ~13 times; and rungs of 3 and 8 eras cannot separate two MA periods at all. The 2026-08-01 run proves it: every finalist scored 25.0-25.9% balanced accuracy - below the 33.3% one-class floor, i.e. indistinguishable noise - and the search then "deployed the winner" of that. THE ERROR WAS THE SCORING FUNCTION, not its constants. Using a full training run to choose a feature's period is a wrapper method paying wrapper prices for a decision that does not need one. The reference book does not do this: ch. 3.3 selects inputs by measuring each candidate indicator's CORRELATION with the target and dropping the ones with none, with no network involved. So: rank candidates by the MUTUAL INFORMATION between the resulting feature vector and the triple-barrier label. MI rather than correlation because the label is 3-class categorical and the features are not monotonically related to it. Equal-FREQUENCY binning (rank-based), because these features are ATR-normalised and heavy-tailed - fixed-width bins put nearly everything in one bucket and report ~0 information for a genuinely useful feature. Scoring is arithmetic over the feature cache, so it costs seconds and its cost is independent of topology: LSTM now tunes as fast as the MLP. Coordinate sweep, not product sweep - cost is the SUM of per-parameter candidate counts, so enabling every indicator stays affordable - with a second pass that breaks early once nothing moves. Sampling is IS-ONLY. Letting the OOS window influence which indicator settings ship would mean the holdout had been used for selection and had stopped being a holdout. HONEST LIMIT, recorded because it is the price: MI is marginal, so a parameter that only pays off in combination with another can be missed (Guyon & Elisseeff 2003, filter vs wrapper). Given the wrapper it replaces was ranking pure noise at 48 h a run, this is strictly better. Deleted with it: GaRungEras/GaExtract/GaStore/GaMutate/GaRandomCandidate/ GaBlockCrossover/GaSortAliveByScoreDesc/GaBreedNextGeneration, 14 m_ga* members, the GA_*/TUNE_POP_* constants, and ComputeTuneTrialBudget. AND m_evalMode/m_evalEraBudget, because nothing set them any more - 28 read sites all permanently inert. That is not a tidy-up: the `if (!m_evalMode)` guard on UpdateClassPriors is exactly what silently disabled the imbalance correction for entire runs two commits ago. Dead machinery that still reads like live machinery is this codebase's most expensive recurring bug, and leaving 28 more instances of it would have been indefensible. The panel's tuning-progress state goes too - tuning no longer takes long enough to need one. Both builds compile 0 errors / 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:29:31 -04:00
//--- the MQL5 RNG state. Matches the tuner's own post-tune rebuild path
//--- (line ~5071) and Warrior_EA.mq5's OnInit.
MathSrand(GetTickCount());
bool rebuilt = BuildFreshTopology();
if(!rebuilt)
Print(ID + ": ERROR - failed to rebuild fresh topology after weights reset");
else
{
//--- Stamp the config fingerprint with isInitialized=FALSE, NOT m_isInitialized. The compare in
//--- InitNeuralNetwork (and its own fresh-start .cfg write) always run before m_isInitialized is
//--- set true at the end of init, so they always use false. ResetWeights is invoked from the panel
//--- AFTER init, where m_isInitialized is true - passing it here would make this the ONLY .cfg on
//--- disk with isInitialized=true, so the very next attach spuriously fails the compare
//--- ("Configuration mismatch. Deleting file") and needlessly discards the weights we just reset.
//--- This flag is runtime lifecycle state, not a topology/input parameter, so it must never gate reuse.
feat(nn): derive dense depth, train on all history, pin the shape in .cfg Completes the derived-topology work. Three inputs removed. AIType loses its depth suffix - AI_MLP/AI_CONV/AI_LSTM/AI_HYBRID, five entries instead of eight. Depth is now derived from the two endpoints the taper already has to connect (derived first-layer width, output-tied final width) at a 2x per-layer compression target, clamped [2..5]. Asking a user to pick a layer count while the code derives the widths those layers taper between was asking for half a decision: at 64 units tapering to 12, four layers compress by 1.4x per step and five by 1.3x, so the extra depth bought no abstraction. On the shipping H1/10y default the derivation lands on 3 layers - the depth that actually won Run 2. StudyPeriods removed. There is no case for training on less data than the broker provides at a ~6% directional base rate; the honest generalization read comes from the OOS holdout, not from withholding history. Training now starts at the earliest available bar, floored by MinTrainYear, which answers a different question (excluding dubious pre-history) and stays. That required closing the hazard the old code documented: the capacity budget now MEASURES the symbol's real bar count, and a topology derived from a measurement would widen as history downloads. Both ends are now pinned. Every derived value left the weights-filename fingerprint - keying a filename on a measured quantity means the EA looks for a file that does not exist, starts from era 0 and orphans a trained model, silently, because a missing cache is the normal first-run state. The shape lives in the .cfg instead, where LoadAndCompare now ADOPTS the four derived fields rather than diffing them; a mismatch there would discard a fully-trained model over nothing the user did. Two fields appended to the .cfg for the conv/LSTM stages, length-guarded on read because FileReadInteger past EOF returns 0 with no error. ForceHiddenLayers, a compile-time constant like DebuggingMode, pins depth for diagnostic comparisons. It joins the fingerprint only when non-zero, so forced depths get their own files - sequential comparisons only, not simultaneous from one .ex5. Derived shape, H1/10y defaults (21 features x 20 bars): first layer 64, 3 dense, 8 conv filters, 16 LSTM units. The LSTM block halves from ~58k to ~28k weights. Both builds compile 0 errors, 0 warnings. Re-keys existing models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 10:05:40 -04:00
SaveTopologyConfiguration(m_activeFileName, m_initialNeuronsCount, m_hiddenLayersCount, m_neuronsReduction, m_minNeuronsCount, m_optimizationAlgo, m_historyBars, m_outputNeuronsCount, m_neuronsCount, LEGACY_STUDY_PERIOD_SLOT, m_minTrainYear, false, LEGACY_CONVERGE_WR_SLOT, m_fractalPeriods, m_convFilterCount, m_lstmHiddenSize, m_activeFileCommon);
Print(ID + ": weights reset - training will restart from era 0 (current config only: " + m_activeFileName + ")");
}
m_trainingStopRequested = stopped;
if(!stopped && !bEventStudy)
bEventStudy = EventChartCustom(ChartID(), 1, 0, 0, "Reset");
return rebuilt;
}
};
//+------------------------------------------------------------------+
//+------------------------------------------------------------------+
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//| IMPLEMENTATION |
//| |
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//| CExpertSignalAIBase's method bodies live in these partial files. |
//| They MUST be included here, after the class declaration above, |
//| and nowhere else. Order between them does not matter - they are |
//| all out-of-class definitions of an already-declared class. |
//| |
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
//| The class was ~8200 lines in one file; this splits the bodies by |
//| responsibility so a change to, say, chart drawing no longer means |
//| scrolling past the era loop. Nothing was rewritten in the move. |
//+------------------------------------------------------------------+
#include "AIBase\Training.mqh"
#include "AIBase\Lifecycle.mqh"
#include "AIBase\Topology.mqh"
refactor: split CExpertSignalAIBase implementation by responsibility ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 00:42:45 -04:00
#include "AIBase\Labels.mqh"
#include "AIBase\OnlineLearning.mqh"
#include "AIBase\AutoTune.mqh"
#include "AIBase\Inference.mqh"
#include "AIBase\Persistence.mqh"
#include "AIBase\ChartUI.mqh"
#include "AIBase\Features.mqh"
feat: excursion-size head (Stage 1, measurement only) Direction is closed - normalised asymmetry fails on three instruments with a working positive control, and the classifier's own best-of-999 era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is a different question and RANGE clears at ~4x its null. Checked the denomination before building on that, since the source memo warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is predictable" is a claim about travel RELATIVE to current ATR, not a restatement of "ATR is autocorrelated". It is exactly the part a fixed multiple (stop 3.31*ATR, target 1.64*ATR) discards. A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches ladder rung k) upward and downward. Survival parameterisation rather than regressing the multiple, because it needs nothing new from CNet: sigmoid outputs and the per-neuron delta the `total != 3` branch already applies (a quantile head would need a linear activation and a pinball gradient in Network.mqh, Network.cl and the DirectML path, on a class four topologies share). Targets are free - m_ladderUpAt already records first-touch age per rung with 0 meaning never reached. Separate net, not extra outputs on the classifier: more outputs would change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push the count off 3 - the exact condition backProp uses to select the joint softmax gradient the 3-class head depends on. The classifier is bit-for-bit unaffected and this is removable without trace. STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the constant per-rung base rate - the baseline a fixed ATR multiple already assumes - with both predictors fitted IS and evaluated OOS, so neither gets a look at the test set. Positive skill justifies Stage 2 (drive SL/TP and sizing off ExcursionQuantile, which is defined and deliberately uncalled). Zero or negative means ATR already carries everything and Stage 2 must not be built. Trains only on primary occurrences: the replay queue oversamples for CLASS balance, and a direction-balanced sample is a biased SIZE sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 07:40:01 -04:00
#include "AIBase\Excursion.mqh"
//+------------------------------------------------------------------+