forked from animatedread/Warrior_EA
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
AnimateDread
|
94019f363e |
feat(gate): grade OOS calls on the exit policy actually in force, and move vote combining out of the members and into the orchestrator
Option (a) from the exit-policy question: the certified number must be the traded number. Plus the modularity correction the user called for on |
||
|
AnimateDread
|
778b6c09c6 |
feat(gates): derive the recall floor instead of configuring it, publish what the configuration can PROVE, and stop an arbitrary member driving live exits
Three changes, all from the same principle: measure what is there before aiming
at it, and never certify a number you do not trade.
1. THE RECALL FLOOR IS DERIVED, AND IT MOVES BELOW CHANCE.
MinRecall=40 was a constant doing a statistical job. Its reference point is the
33.3% recall a zero-skill 3-class model gets on EVERY class, and against that the
constant was accidentally calibrated for exactly one sample size: on USDJPY CONV
(n_eff 195) 40% is chance + 2.0 SE; on SP500 PAI (n_eff 42) the same 40% is chance
+ 0.9 SE. One chart was being held to a bar twice as strict as the other, for no
reason anyone chose.
CollapseRecallFloorPct() computes it per class from that class's own effective
sample - EffectiveSampleSize(), so the overlap deflation the rest of the gates use
applies here too - as chance - EDGE_MIN_SIGMAS x SE. 26.5% at n_eff 195, 18.8% at
n_eff 42.
BELOW chance, deliberately, and this is the substantive change rather than the
arithmetic. This gate's only job is refusing to call a COLLAPSED model converged.
It is not a quality bar; the deploy gate is the quality bar and it is already
rigorous (chance + 2 SE on the deflated sample, Sidak over candidate eras, then the
cross-instrument pooled certificate). A convergence gate that ALSO demands
provably-above-chance recall on all three classes double-counts that job, and it
has failed that way twice here: MinRecall=60 blocked every SP500 H1 run in 2026-07,
and the 40 that replaced it made Neutral structurally unreachable once first-touch
resolution cut Neutral to a 0.65% residue. A floor nothing can reach does not make
a funded account safer, it stops the run converging at all.
Testing significantly BELOW chance instead catches what a fixed 40 was actually
catching - a model that has stopped emitting a class - and cannot become
unreachable by construction. It also fixes the direction the old constant scaled:
it now widens on a thin OOS window, where low recall genuinely cannot be told from
noise, and tightens on a rich one. Today's SP500 PAI (Buy 51 / Sell 18 / Neutral 30)
is still correctly blocked on Sell.
This also resolves a standing contradiction the code half-admitted at the
isBetterEra comment: selection ranks on coverage-weighted PRECISION while
convergence gated on RECALL, so a sparse high-precision abstainer - precisely the
model that could clear the deploy bar - was blocked by the floor.
The era line now PRINTS the derived floor. Anyone comparing these recalls against a
remembered "40" is reading the wrong bar.
2. DETECTABILITY: WHAT THIS CONFIGURATION COULD PROVE, BEFORE IT TRAINS.
The DEPLOY BAR line states the bar. It never said what reaching it would take, and
that is the actionable direction. ReportDetectability() inverts the same identity -
the gate passes when edge >= z x sqrt(p(1-p)/n_eff), so certifying an edge d needs
n_eff >= z^2 p(1-p)/d^2 independent calls, hence L times as many raw ones - and
prints a +2 / +5 / +10pp ladder as required independent calls, raw calls, and share
of the OOS window, marking any rung that needs more than the window holds IMPOSSIBLE.
Every term is a property of the CONFIGURATION - geometry via break-even, horizon via
mean label lifespan, window via oosCutoff - so no amount of training moves any of
them. It fires once, at the first healthy sweep, beside ReportFeatureHealth, for the
same reason: that is the first moment the bar grid, the measured geometry and the
lifespan are real numbers rather than defaults. It gates nothing.
This is the EdgeFinder discipline applied to our own gate: establish what the market
and the measurement design have to offer, then point the net at it - rather than
spending a thousand eras chasing something this OOS window could never certify.
3. AN ARBITRARY MEMBER WAS DRIVING LIVE EXITS AND TRAILING (user-identified).
Every ensemble member ran
g_LiveAISignedConfidence = SignedAIConfidence();
unconditionally, every tick. Last writer wins. Its consumers are the AI early-exit
route (CExpertSignalCustom::LiveSignedConfidence) and TrailingIntelligent - so on a
four-model chart an LSTM entry could be closed, and its stop moved, on the
Perceptron's opinion alone, decided by scheduling order. Not the vote, not a
weighted blend.
Now the mean across registered members, matching how the ensemble actually trades:
the open decision is the weighted-average vote, and an abstaining member contributes
0 and dilutes exactly as it does there. Members still training read 0, so a
half-trained ensemble reads WEAKER rather than louder - the safe direction for an
exit trigger. Deployed and paused members are included, which is the opposite of the
era barrier's exemption rule and correct for the opposite reason: that one asks who
must be waited for, this asks who has an opinion.
Latent today and staying that way for now by choice - Min_Vote_Close ships Disabled
(101, unreachable on both scales it drives) and TrailingStrategy is off, so live
exits are SL/TP only and the certified hold-to-barrier win rate is what actually
gets traded. Fixed now precisely because the plan is to enable vote exits once the
models are accurate, at which point a scheduling-order exit would be both harmful
and very hard to see.
STILL OPEN, and needs a decision before vote exits go on: the member gate and the
ensemble vote gate both grade hold-to-barrier, so enabling vote exits makes the
certified number stop describing the traded one. Warrior_EA.mq5 currently argues
barrier models may keep vote exits because "their label IS the vote's own horizon" -
that does not hold, since a vote flip at bar 5 of a 64-bar horizon is not the
target-before-stop outcome the gate measured. Either grade the OOS call on the real
exit rule (first of SL / TP / vote-flip / horizon) through the fill engine, or set
HoldToBarrier for ensemble members so the policy cannot drift from the certificate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
AnimateDread
|
be396749fc |
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at
Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with
four of their eight members completely silent. Nothing in this commit guesses at
why the sweep fails - the last five guesses were all wrong. It makes the failure
say what it is, and stops one broken member taking its whole chart down with it.
WHAT THE LOG ACTUALLY SAYS, before any of this.
- The running build IS
|
||
|
AnimateDread
|
45c9e211b3 |
feat(depth): prime -> settle -> sweep, and name which handle is short
"Max bars in chart" is set to Unlimited, so the static-terminal-limit reading in |
||
|
AnimateDread
|
7e63a8be01 |
fix(depth): route EVERY ResizeBuffers call site through one indicator-depth gate
|
||
|
AnimateDread
|
1dda479261 |
fix(train): clamp the sweep to indicator-servable depth - the scan wall was CopyBuffer, not a cold indicator
Symptom: on a 3-chart run with contention ruled out (SP500 sitting at era 2552),
USDJPY and XAUUSD produced 0 usable windows out of 50,163 and 33,966 - forever,
re-sweeping on every discard, which is the panel oscillating 0->100%.
Bars() is the PRICE series depth. A CUSTOM indicator's is not: MT5 calculates it
in its own context bounded by "Max bars in chart" (TERMINAL_MAXBARS), and
CopyBuffer past that limit does not short-read, it FAILS - so CDoubleBuffer keeps
nothing and EVERY index answers EMPTY_VALUE. ADMovingAverage is the only custom
indicator whose feature block REJECTS on EMPTY_VALUE (ADZigZag, also CiCustom,
neutral-fills; RSI/MACD/Ichimoku/ATR are built-ins served at any depth), so the
sweep died on feature 25 of every bar while the 24 price features under it were
fine. That is exactly the "window had 24 of 832 values" the stall report named.
Perfectly depth-correlated, measured 2026-08-17:
SP500 16,234 bars -> era 2552 XAUUSD 33,982 -> 0 windows
XTIUSD 16,611 bars -> era 71 USDJPY 50,179 -> 0 windows
This RETIRES the 2026-08-17 cold-indicator reading of the same stall.
|
||
|
AnimateDread
|
1cf4c57d57 |
fix(altdata): median-fill instead of zero-fill, and a one-shot feature-vector autopsy
ALT-DATA AUDIT. The files themselves are healthy - all six symbols, 6,073 daily
rows, 2010-01-01 to 2026-08-17, no constant or degenerate columns, sane tails
(mac_cpi/mac_unemp flat ~47d is monthly data behaving correctly). The problem is
not the data, it is what happens where the data ISN'T.
CAltDataPanel::Features() returned an all-ZERO vector for any bar older than the
file's first row, and left blank cells at 0 too. Both were deliberate ('the block
is additive context and must degrade, never reject the bar') and that reasoning
holds for the CHANGE columns - but half these features are LEVELS: vix, ivol,
mac_y10, mac_cpi, mac_unemp, eia_util. For a level, 0 is not a missing reading,
it is an impossible one far outside the series' range. VIX does not visit zero.
And the spike lands in exactly the wrong place. Every alt file starts 2010-01-01
while the charts run far deeper - USDJPY H4 reaches ~1994, roughly HALF its
history - so 'alt block is all zeros' is precisely the predicate 'this bar is
older than 2010'. The IS/OOS split is chronological, so that predicate covers
~half of IS and none of OOS: an in-sample feature guaranteed to be useless
out-of-sample, and a bimodal input for the first BatchNorm to normalise. Not a
lookahead leak - a distribution corruption, which is quieter and was never
reported anywhere.
Now filled with the column MEDIAN over the covered range. A constant cannot leak
whatever its source - it takes the same value on every pre-coverage bar, so it
carries no information about which of those bars won - which is what makes a
median computed over later data legitimate here. Median not mean because the
series are skewed. Blank cells get the same treatment (eia_stk_idx1y alone has
181 blanks in 6,073 rows) and the count is now logged at load.
THE BACKOFF WAS ALREADY THERE AND WAS DEAD. Training.mqh arms m_coldSweepTick on
m_featureFailTransient, but only the open/ATR guards ever set that flag, so
f0cf659's cold ADMovingAverage looked PERMANENT and the sweep re-ran at full
speed forever. Setting the flag in the indicator guards revives the mechanism
that was already designed for this; no second backoff was needed and the one I
first wrote has been removed in favour of it.
SELF-HEALING, as asked. ReportFeatureHealth() runs once, the first time pass 1
produces usable windows, samples ~400 bars spread across the whole training range
and names every feature slot that is CONSTANT or mostly-zero, tagging alt-block
slots as alt[i]. Both of today's failures were the same shape - a block silently
produces nothing while every downstream number stays plausible - and neither an
accuracy figure nor a model can tell 'this feature is always 0' from 'this
feature is genuinely 0 here'. Evenly spaced sampling so a block that dies only in
deep history is caught as surely as one dead everywhere. A report, not a gate:
a rare-flag feature can be legitimately constant, and refusing to train would
turn a diagnostic into an outage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
AnimateDread
|
f0cf659945 |
fix(features): a cold indicator is TRANSIENT, not a permanent miss - and back off instead of re-sweeping
Six fresh instances on USDJPY and XAUUSD swept 33,965-50,162 bars and produced
ZERO usable feature windows, repeatedly, for 40 minutes and 239 stall reports,
without ever completing era 0. The four instances already warmed up before those
charts were attached trained normally throughout.
THE STALL REPORT NAMED THE SPOT EXACTLY: 'lookback slot 0 REJECTED the bar
(window had 24 of 832 values)', and 24 is the core block to the value - 4 price +
5 swing + 4 range + 4 volume + 6 time + 1 ATR. So feature 25 was the wall, and
feature 25 is the first value of the MA block. The same 24 appeared on XAUUSD
against a 816-value window (51 features/bar vs 52), which is what ruled out any
symbol-specific data gap: the wall sits at a fixed feature index, not a date.
ADMovingAverage is a CUSTOM indicator, so MT5 fills its buffer asynchronously and
returns EMPTY_VALUE for EVERY index until it has calculated - not just the
warm-up tail. That guard did not set m_featureFailTransient, so every bar of the
sweep was cached as a PERMANENT miss. This is precisely the failure the ATR guard
twenty lines above it was fixed for on 2026-08-10; the fix was never propagated
to the indicator blocks that follow. RSI, MACD and Ichimoku had the same defect
and are fixed too. (The Donchian high/low guard is a break into a
degraded-but-usable path, not a rejection, and is deliberately left alone.)
IT ALSO SELF-SUSTAINED, which is why it never recovered. The ok=0 self-heal drops
the feature cache and re-sweeps immediately, so each stuck instance spent every
millisecond re-reading 30-50k bars - six of them at once, on a six-core box,
competing for CPU with the very indicator calculation they were all waiting on.
The recovery was preventing the recovery. A transient total failure now re-arms
m_warmupPassesRemaining, yielding the CPU for a few separately-scheduled Train()
calls - the same mechanism a fresh model already uses to let history sync finish,
pointed at indicator warm-up instead.
Verified in the terminal journal first: indicators load and unload in matched
counts and there is no OOM, so this is NOT the
|
||
|
AnimateDread
|
d87f7d88ff |
feat(gate): cross-instrument pooled certification
The deploy bottleneck is CERTIFICATION, not training. A 4,738-bar OOS window at L=75.6 holds ~63 independent observations; certifying a 3pp edge at 2 sigma needs ~1,036. More bars of the same symbol barely help - they overlap. Other symbols do not. WHAT POOLS. Not win rates: symbols have different derived geometries, different break-evens and different drifts, so averaging raw rates across them is meaningless. What pools is each symbol's EXCESS OVER ITS OWN CHANCE RATE, combined by inverse-variance weighting (fixed-effects meta-analysis). Each symbol keeps its own model, geometry and chance rate; only the evidence is combined. THE CORRELATION PROBLEM, bracketed rather than assumed away. SP500 and NAS100 are ~0.9 correlated and pooling them as independent inflates the evidence. Nothing here can measure that without sharing return series, so instead of guessing a correction the gate reports both ends: SE_INDEP = sqrt(1/SUM(1/var_i)) all members independent SE_CORR = SUM(w_i * sqrt(var_i)) all members perfectly correlated The truth is always between. THE GATE USES SE_CORR, so a pass cannot be an artifact of correlated instruments - that bound already assumes the worst. The ratio is logged as the diversification credit the gate declines to claim, so the cost of that conservatism is visible instead of hidden. SCOPE, deliberately limited: the pooled result is REPORTED, never folded into tradeableOK. The local gate certifies the model that actually trades this symbol; the pool answers the different question of whether the strategy has an edge at all. Letting a cross-symbol result license a local deploy would ship a model that never cleared its own bar - so it cannot. Mechanics: one file per instrument (no concurrent-write path to get wrong), every FileOpen carrying FILE_SHARE_READ|FILE_SHARE_WRITE, records skipped rather than reinterpreted on a version mismatch, 12h staleness cutoff so a stopped chart cannot vote, and pooling refused below 3 instruments. Poolability requires matching timeframe and ratio; differing SYMBOL is the entire point. Publishing is unconditional - a pool that only hears from winners is a selection effect, not a meta-analysis. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
2b5d0f8355 |
feat(measurement): fix zero-skill denominator, publish the deploy bar, measure lifespan per rung, add a MEASURE scale objective
The last run could not have demonstrated an edge either way, and nothing in the
log said so. Four changes so it does.
1. THE ZERO-SKILL LINE DIVIDED BY THE WRONG DENOMINATOR. m_oosWinLongTotal resets
every era; m_oosSamples only resets on a full model reset. So 'always-long %'
decayed as ~1/era: a run whose true rate is 37% printed 1.2% at era 33 and
0.0% at era 2219. This is the SAME bug already found and fixed for
logBuyPredPct thirty lines above ('era-15 Buy:2% that was really ~30%'), left
in the one line whose whole job is to be the reference every other number is
read against. Correct at era 1, wrong everywhere after - including the '62%
zero-skill' figure in the 2026-08-16 notes. Now per-era, and always-short is
finally readable.
2. THE DEPLOY GATE STATES ITS OWN BAR. 'edge -1pp' era after era cannot separate
'short by a hair' from 'short by an amount no strategy could cover'. The era
line now prints the required win rate, the SE, the effective n and the
lifespan it was deflated by; above 100% it says UNREACHABLE. At 4,738 OOS bars
and L=75.6 there are ~63 independent observations, putting the bar near 66% at
typical coverage.
3. LIFESPAN MEASURED PER RUNG. The first-passage cache already stores touch ages
at every ladder level, so each candidate geometry's resolution time is
readable without training on it - L-vs-width becomes a measurement across the
whole ladder in ONE run rather than a second chart. Each rung reports L,
n_eff, min provable edge and min provable EV.
4. SCALE OBJECTIVE IS PHASE-AWARE, defaulting to MEASURE. Width and detectability
are opposed: labels overlap by L, L grows like m*k = width^2 at fixed ratio,
so min provable EV ~ width^2 while the cost saving from width is only linear.
Doubling width quadruples the smallest EV you can prove. DEPLOY (widest that
clears reachability) is right once an edge is known; MEASURE (narrowest that
keeps round-trip spread under BARRIER_MAX_COST_FRACTION_PCT) is right while it
still has to be shown. The direction does not depend on the exponent, and
item 3 makes the exponent checkable.
Fixed in review: m_lastRungLifespan is cleared on every LadderWinShare entry or a
rejected rung reports the previous rung's lifespan as its own; per-rung
detectability is labelled IS-sample based (the deriver may not see the holdout),
so absolute figures are optimistic while the ranking is unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
AnimateDread
|
1540ba8e64 |
fix(labels): overlapping-label sample correction + horizon cap on the scale ladder
Three defects, all surfaced by the 2026-08-17 SP500 H4 run that shipped
stop 4.86 / target 9.71 (width 14.57*ATR, horizon 384).
1. EVERY STANDARD ERROR ASSUMED INDEPENDENT SAMPLES. Triple-barrier labels
started one per bar overlap by the label's lifespan, so n calls are worth
~n/L independent observations (Lopez de Prado, AFML ch. 4 - sample
uniqueness). All three sqrt(p(1-p)/n) sites divided by the RAW count.
The tell: the operating point's null-of-the-maximum gate is family-wise and
should fire on ~5% of eras under the null. Measured fire rates - PAI 47/73
(64%), ConvLSTM 9/24, LSTM 8/21 (38%), CONV 4/62 (6%). CONV, the only model
whose margin distribution admits few bins, sat on the null; the rest cleared
a bar that was too low by ~sqrt(L). PAI's deployed threshold consequently
alternated between the ENDS of its own range era to era (0.10 -> 0.88 ->
0.86 -> 0.66; coverage 16% <-> 73%).
TripleBarrierLabel now records when each label became KNOWABLE - the first
winning touch, or both stops, or the timeout - and the prebuild accumulates
the mean. EffectiveSampleSize() feeds the operating point, the member deploy
gate and the ensemble vote gate. Conservative by construction (n/L is an
upper bound on the damage); gates get harder, never easier.
2. THE SCALE LADDER RAN AWAY, again. Horizon scales as swingMedian*sl*tp, and
since
|
||
|
AnimateDread
|
bc57aca15d |
fix(geometry): the target was small BY CONSTRUCTION - ratio is now policy, scale is measured, ladder ceiling removed
The derivation read the stop from q75 of ADVERSE travel and the target from q50
of FAVOURABLE travel. Over one horizon those distributions are broadly the same
shape, so q75 > q50 MECHANICALLY - the target came out smaller than the stop no
matter what the market did. SP500 H4 shipped stop 3.07 / target 1.70: a 0.55:1
payoff needing 64.3%. That was never a measurement, it was two mismatched
constants.
The reachability line printed beside it - "target on 50.0% of bars, stop on
25.0%" - is exactly 1-q50 and 1-q75. Tautological. It cannot disconfirm
anything, and it read as validation.
WIDTH AND RATIO ARE INDEPENDENT AND ONLY ONE PAYS. EV = edge x width;
ratio is EV-neutral (a driftless walk reaches +m before -k with probability
k/(k+m), which IS break-even). Width is what buys cost efficiency: the spread
is a fixed 0.047*ATR here, so the shipped 4.77*ATR width paid it 21 times per
unit of travel. So:
RATIO = policy. BARRIER_TARGET_RR = 2.0 (user's 1:2). Break-even 33.3%.
SCALE = measured. The stop quantile is chosen from a ladder, WIDEST FIRST,
taking the first rung whose implied 2x target is still reached often
enough to be a trainable class.
That last clause is the difference from the min-reward:risk raise removed in
2026-08-09, which forced target = 2 x stop with NO reachability test, landed on
6.66*ATR reachable on 3.3% of bars, and trained the model to predict something
that essentially never happened. Same ratio; the scale now retreats until the
data says the target is attainable. Every rung is logged.
LADDER CEILING REMOVED. BARRIER_LADDER stopped at 5.00 and the expectancy scan's
"best resolvable pair on width alone" came back as stop 5.05 / target 4.95 - it
pinned to the top rung. A recommendation landing exactly on the edge of its own
search space is a boundary, not a finding: it cannot tell "5 ATR is optimal"
from "5 ATR is all we allowed". Extended to 20*ATR (8 -> 14 rungs). Nothing else
needs editing - every consumer is parameterised by BARRIER_LADDER_COUNT - and
the horizon constraints (decided >= 60%, reachability floor) now bind instead of
a constant.
THE SCAN COULD NOT SEE THE SHIPPED GEOMETRY. ReportBarrierGeometryScan looked
the configured pair up in its integer grid, and DeriveBarrierGeometry produces
CONTINUOUS multiples (3.07/1.70) that can never equal a grid point - so
cfgExcess stayed at its -1.0 sentinel and the report printed "configured 3:2
scores -1.00000", which reads as a catastrophic score and actually means "never
evaluated". Worse, the grid skipped target<stop entirely because it "inverts the
trade's whole premise" - while the derivation was shipping exactly that. The
incumbent is now always scored as a peer (never crowned; it is already in force
and is not an enum pairing the scan could adopt).
BREAK-EVEN NOW INCLUDES THE SPREAD. Every report quoted the frictionless
SL/(SL+TP). On SP500 H4 that read 64.3% while the MEASURED zero-skill rate was
62.1% - a 2.2pp gap that IS the cost, and that made every model look 2.2pp
better than it was. CostAdjustedBreakEvenPct() prices a win at (TP - spread) and
a loss at (SL + spread), matching the expectancy scan's convention exactly so
the two reports cannot disagree.
It also feeds FitDirConfThreshold, which is the correctness half: the operating
point subtracts break-even from precision, so the frictionless figure made every
candidate threshold look better by the width of the spread - 2.2pp against a
measured edge of 2.3pp, i.e. very nearly all of it.
Era line now carries both: "break-even 64.3% frictionless, 66.6% AFTER SPREAD".
Forces a full relabel and retrain. Requested.
NOT COMPILED - user compiles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
AnimateDread
|
d30420e3f2 |
fix(batchnorm): bound the normalized value - a constant input feature was amplified 1e4x and pinned PAI's head to its rails
BN_MIN_STD = 1e-4 caps the per-unit gain at 1/1e-4 = 1e4, and the comment above
it states that as though it were a safety property. It is not. A unit whose
running variance is ~0 is a CONSTANT feature carrying no information, and
dividing its rounding noise by 1e-4 hands the next layer an activation of
several hundred. BN's contract is "output has ~unit variance"; a unit that
cannot supply that must contribute nothing, not the largest signal in the layer.
MEASURED, 2026-08-17 SP500 H4, four topologies on identical separate charts:
model spread Neutral CHOSE Neutral TIED rail
CONV 0.386 0.68% 0.10% 0.48%
LSTM 0.392 0.63% 0.00% 0.00%
HYB 0.376 1.79% 0.00% 0.01%
PAI 0.192 0.09% 80.63% 99.99%
bn1's cached nx normed 1.38e4 over 800 units. PAI's SIGMOID head was on its
rails on 99.99% of bars, with Buy and Sell landing on the SAME rail so they
compared exactly equal, and ApplyClassificationSoftmax()'s strict-majority rule
reported that tie as Neutral on ~80% of bars.
So the long-running "PAI is heavily biased toward Neutral" was never a
class-prior problem: the net CHOSE Neutral on 0.09% of bars. It was float
equality on a saturated head. The
|
||
|
AnimateDread
|
331ab29c56 |
feat(diagnostics): split a reported "Neutral" into CHOSE vs TIED - they need opposite fixes
ApplyClassificationSoftmax() requires a STRICT majority over both rivals and
sends every tie, 2-way or 3-way, to Neutral. So "OOS recall Neutral:100%" is
two completely different events sharing one label:
CHOSE - the net genuinely ranks Neutral highest. A class-prior/label problem.
TIED - the top two are EXACTLY equal, so the net expressed no preference and
the tie-break reported Neutral. A SATURATION problem: the head is
SIGMOID, and a saturated sigmoid returns exactly 0.0f or 1.0f in the
DLL's float32, so two classes pinned to the same rail compare equal
and the bar is silently discarded.
Nothing in the logs could tell them apart, and the fixes point opposite ways.
Eras 1-25 of the 2026-08-17 solo PAI run read "Neutral 100%" at spread avg 0.99
- fully saturated - and broke out at era 27 as the spread fell to 0.75. That is
consistent with EITHER story. The user reports the Neutral phase on most runs,
so it is worth four longs to stop guessing.
Four per-era counters on the pass 3 OOS walk, reported as:
| Neutral CHOSE 12.4% / TIED 38.1% (of which B=S 1204) | rail 61.2%
m_oosNeutralStrict - Neutral strictly highest
m_oosNeutralTie - no strict winner; the tie-break produced Neutral
m_oosTieBuySell - the costly subset: Buy and Sell tied AT the top, i.e. a
DIRECTIONAL reading thrown away by float equality
m_oosRailBars - any raw output sitting on a sigmoid asymptote, the
saturation that makes exact ties possible at all
Read on the RAW logits, before ApplyClassificationSoftmax() overwrites TempData
in place. Legitimate because softmax is strictly monotone: it cannot change the
ordering and cannot break a tie either, so the raw reading and the decision
always agree. Placed alongside the existing min/max/spread capture so all the
output diagnostics describe the same values.
Measurement only - no decision path reads these.
NOT COMPILED - user compiles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
AnimateDread
|
208da4cbaa |
fix: drop the ranking slice for the calibration band; un-collapse the tiers
NOT COMPILED - user compiles. (1) THE RANKING SLICE IS GONE. It reserved 20% of the OOS window so the pattern-DB backfill would read bars the deployed checkpoint was not SELECTED on. That objection stands; carving a new region to answer it did not. The calibration band already has every property the slice was buying: never trained on | never graded by pass 3 (which walks [0, oosCutoff) and so never reaches it) | never seen by the deploy gate | purged by a full label horizon on BOTH sides | and larger besides - 1,684 bars vs the ~970 carved So the backfill now walks [calibLo, calibHi) and pass 3 goes back to grading the entire OOS window, exactly as before any of this. The gate gets its full sample back (~10% of a sigma), the split loses a region, and the failure mode found an hour ago - a reserved region silently blanking ~10 months of chart arrows, because arrows are only drawn on bars pass 3 grades - becomes impossible. One impurity, stated in the completion log rather than hidden: m_dirConfThreshold is FITTED on that band and the walk applies it to decide which bars fired, so coverage there is mildly optimistic. One scalar under a coverage floor, against checkpoint selection over hundreds of eras. This backfill IS the deploy-time warm-up: it runs right after FinalizeTrainRun() restores the deployed weights, so it scores with exactly what is about to trade. (2) EVERY CALL WAS TIER 0, AND IT WAS ARITHMETIC. ConfidenceTier() quartiles [floorConf, 1] where floorConf = 1/3 - the lowest magnitude a 3-way softmax winner can hold. But it was fed CalibratedConfidenceMagnitude(), which multiplies by m_confidenceCalScale, clamped to [0.3, 1.5]. That lower clamp is BELOW 1/3. Whenever calibration bottoms out, t goes negative and MathMax(0, ...) pins every call to tier 0. Which is what the live run does. m_confidenceCalScale is EMA'd toward empiricalAccuracy / avgClaimedConfidence; with the model over-calling Neutral, 3-class agreement sits near 10% against a claimed confidence near 0.9, so the ratio is ~0.11 and clamps to 0.3 every era. Logged: tier prec T0:72%(828) T1:n/a(0) T2:n/a(0) T3:n/a(0) 828 calls, one bucket - the four tier weights and the entire per-tier pattern-DB ranking reduced to a single number. The backfill was feeding a mechanism that structurally could not rank. Tiering now reads the RAW head magnitude, which genuinely lives on the [1/3, 1] range these bounds were written for. Calibration keeps its real jobs - AIConfidence() for MM sizing and SignedAIConfidence() for the vote are unchanged. STILL OPEN, deliberately not touched here: the calibration TARGET itself. empiricalAccuracy is 3-class agreement, which is the wrong quantity to scale a DIRECTIONAL confidence against - it counts a Neutral class that is 0.19% of labels. The honest target is the win rate on the calls the confidence describes (directional precision), with the claimed-confidence average taken over those same called bars. That needs a new accumulator and it interacts with the Neutral over-calling being fixed elsewhere, so it wants one clean run first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
75d23e9b82 |
fix(gate): move the ranking slice to the OLD end - it walled off the recent chart
NOT COMPILED - user compiles. User: "there is quite some trading going on, but absolutely nothing on the recent area of the chart, like there is a hard wall starting around november 2025." That wall is 7caf2f6's ranking slice, and it was placed at the wrong end. Chart arrows are only ever drawn on bars pass 3 GRADES, and the slice reserved the NEWEST 20% of the OOS window plus a label-horizon purge. At the live sizing - ~4,860 OOS bars, 128-bar horizon - that is ~1,100 H4 bars withheld from grading, about ten months back from today, exactly where the wall appears. The invisible cost was worse than the visible one: it handed the deploy gate the OLDEST 80% of the OOS window and withheld the most recent regime from the single decision that has to generalise forward. Both fixed by putting the reserve at the oldest end instead: [0, oosScoreHi) OOS - graded by pass 3 (NEWEST, arrows restored) [oosScoreHi, rankLo) purge - one label horizon [rankLo, oosCutoff) RANKING - backfill only, graded by nobody [oosCutoff, calibLo) purge [calibLo, calibHi) CALIBRATION ... IS Of the three consumers competing for those bars, recency is worth least to the ranking: it is an ORDERING of confidence tiers, far less regime-sensitive than an absolute win rate, while the gate's power and the operator's read of the chart both want the newest data. The slice keeps every property that made it worth carving - never graded, never selected on, never seen by the gate, purged on both sides - so the backfilled rows are still honestly out-of-sample. RankSliceHiIndex is replaced by RankSliceLoIndex + OosScoreHiIndex; pass 3 now excludes the slice at the TOP of its walk and descends to 2 as it always did. The backfill walks [RankSliceLoIndex, oosCutoff) via a new m_dbBackfillStopIndex, clamped at both ends so a degenerate slice yields an empty walk rather than one that wanders into graded bars. Verified no reference to the old helper survives. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
1eeed3ac06 |
revert(ui): restore the unconditional era-end arrow repaint
|
||
|
AnimateDread
|
b19799910d |
fix(ui): chart arrows follow the BEST checkpoint, not the latest era
User report: "as soon as the next era training begins the chart signals are
erased, they should persist for as long as they are accurate."
Cause: PruneDirectionalClusters ran unconditionally at the end of pass 3, and it
DELETEs the arrow on any bar the CURRENT era scored Neutral. A model exploring
away from its best therefore wipes the chart every era even though the best
checkpoint still calls those turns. On the run that prompted this the model sat
at OOS recall Neutral:100% for 40+ consecutive eras, so essentially every arrow
was deleted at every era boundary.
The render is now deferred to the era-end block - the first point that knows
whether the era beat the best checkpoint - and only a new best repaints. On eras
that did not improve, the previous best's arrows stay untouched. Two exceptions
keep the chart from ever showing nothing: before the first checkpoint exists
there is no best to preserve, so early eras still paint; and a finishing run
repaints unconditionally, because FinalizeTrainRun is about to restore the
deployed weights and the chart must describe THOSE.
Recorded for ensemble members too. A member's own best era is not the deployable
one (the joint checkpoint decides that), but it is still the most accurate thing
that member has drawn, and the alternative is a chart that empties itself.
Also verified against the log, since two other symptoms were reported alongside:
era cadence PAI-b6b5 (before these changes) 3.76 s/era
PAI-17ae (after) 3.53 s/era
topology both "2 dense from 16 units | input 800 (16 bars x 50)"
calibration both fitted on 1699 held-out bars
So training speed is unchanged - the ~6% is the ranking slice removing 20% of
pass 3's bars. It only FEELS fast because this is a single PAI chart taking the
whole 120 ms budget, not four ensemble members sharing it behind an era barrier.
The Neutral collapse is also pre-existing, not new: b6b5 ran at Neutral 94-100%
with 1-5% directional calls for all 723 of its eras, before any of this work.
That is the known neutral-collapse/recall-gate failure mode, and it is what
"barely drawing signals" actually is. Worth watching, separately: b6b5 reached
best-bal 34.2% by era 723 while 17ae is at 13.6% after 77 - too early to read,
but it is the number to check once 17ae has run comparable eras.
Compile-verified: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
AnimateDread
|
7caf2f626e |
feat: derived taper restored; DB ranking reads a reserved slice, shrunk
TOPOLOGY - reverts the two constants and drops CausalHiddenLayerFloor. The MQL5 article's 30%-per-layer cut and floor of 20 are load-bearing on ITS first-layer width of 1000 (1000->300->90->27 needs a floor to stop). This codebase MEASURES that width, and on the live SP500 H4 config it is 16 units - already floored, with the budget printing "11360 estimated in-sample bars cannot support a 800-wide input ... roughly 1.1 weights per training bar - expect overfitting". At 16 units a floor of 20 makes lastHidden >= m_initialNeuronsCount, so ComputeHiddenLayerCount returns on its first branch and the width taper - the only part derived from this symbol's data - became dead code on all four ensemble members, with depth (2 -> 4) set entirely by counting feature domains. ComputeLayerWidths had already rejected this exact pair of constants in its own comment. The causal floor's premise does not hold either: layers are not inference steps. The "1 layer linear / 2 nonlinear / 3 multi-connected" result is Lippmann 1987 and is about hard-threshold units; with sigmoid/ReLU, Cybenko 1989 and Hornik 1991 give universal approximation from a single hidden layer. Depth buys parameter efficiency for compositional functions, not reasoning hops. ForceHiddenLayers remains for measuring depth directly. RANKING SLICE - the backfill no longer reads the window it is judged on. The deployed checkpoint is CHOSEN as the best-scoring era on the OOS window, so win rates measured back over it are selection-inflated, and the backfill was writing exactly those into the table filter weights rank on: the selection set consumed twice, beside a deploy gate that applies a Sidak correction for that effect. The newest RANK_SLICE_PCT_OF_OOS (20%) of the OOS window, plus a label-horizon purge, is now reserved and graded by nothing - not pass 3, not checkpoint selection, not the gate. The backfill reads only that. The gate keeps ~80% of its measurement (power goes as the square root, so ~10% of a sigma), and the slice is the newest data, which is the regime about to be traded. RankSliceBars returns 0 when no honest slice fits and the backfill then REFUSES and says so, rather than falling back to the scoring window and looking like a success. SHRINKAGE - per-tier win rates are shrunk toward the filter's own pooled rate by MIN_TRADES_FOR_WIN_RATE pseudo-trades before becoming weights. The raw ratio at the minimum sample count carries a ~15pp standard error, so a tier that went 8-2 was handed weight 80 and outranked a tier measured over hundreds of calls at 55 - the ranking was being driven by which small tier got lucky. Opt-in per call site (priorWeight 0 keeps the raw behaviour). Compile-verified: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
b6736fd40b |
fix: the DB backfill could never run, and HEAD did not compile
Four defects in 64c5dd5/1a05e63, found by review + a baseline compile. Goals 1-8 of that session are unchanged; this makes 6 and 8 actually reachable. 1. HEAD DID NOT COMPILE - 6 errors. CControlPanel::Minimize/Maximize were declared `virtual bool ... override`, but CAppDialog declares both as `virtual void` (Controls\Dialog.mqh). errors 265 + 404 on each, plus 151 on `bool ok = CAppDialog::Minimize()`. Return type is void now; there was never a success flag to forward. Verified: 0 errors, 0 warnings. 2. THE BACKFILL COULD NEVER ADVANCE, and neither could the OOS continual simulation (that one has been dead since it was written). Both are armed at the instant convergence is declared, and both advance only from inside Train(), one chunk per call. But ScheduleTrainingIfNeeded's only per-tick ArmStudyEvent site sits in the `else` of a branch taken whenever m_trainingComplete is set and m_trainRunActive is clear - which is exactly the state FinalizeTrainRun() leaves behind one line before they are armed. Train() was never called again, so the walks sat at their start index forever: no "simulation complete" line, and not one row written to the DB this feature exists to fill. Only a manual Resume/Retrain unstuck them. Both flags now keep the model schedulable. 3. IN AI_HYBRID - the mode this ships in - the backfill was never even armed. Ensemble members deploy at Train() ENTRY and return immediately (so no era is wasted), which skips the era-end block the backfill was started from. All four members were a no-op for a second, independent reason. Armed on the ensemble deploy path too, from m_resumeBars/m_resumeOosCutoff. 4. RE-RUNS DUPLICATED ROWS. RegisterSignal inserts unconditionally - no key, no duplicate check - and m_dbBackfillDone is in-memory, so every later attach that retrained to convergence wrote a second full set of rows for the same bars. The ranking would count one bar once per model that ever deployed, weighting superseded opinions as heavily as the live one. A .dbfill marker stamps the deployed era; written only on completion (an interrupted walk redoes itself rather than ranking a partial window) and deleted with the other sidecars on reset-weights. Also: WarmBlocking's timeout was silent, which restored the exact silent pin failure it was added to prevent - it now says so in the journal, and returns true for "no reference pairs to wait for" so the warning stays rare enough to be read. Not addressed, needs a decision: the backfill scores the OOS window with the checkpoint that was SELECTED as best on that same window, then writes those win rates into the table filter weights rank on - the selection set consumed twice, undiscounted, while the deploy gate right next to it applies a family-wise correction for exactly that effect. The rows are also simulated triple-barrier outcomes at today's spread sharing a table with realised fills. The completion log line now states both plainly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
64c5dd55d3 | feat: implement one-shot pattern-database backfill and enhance accuracy tracking for ensemble models | ||
|
AnimateDread
|
e049b624ba |
feat(ensemble): deploy gate on the COMBINED VOTE, with a joint checkpoint
The unit of evaluation in ensemble mode becomes the vote, because the
vote is what trades (user: "at the end of the day they will vote
together during live trading so that would make sense").
Four decisions move from the member to the ensemble:
* which era is "best" -> the era whose COMBINED VOTE scored best
* what is checkpointed -> a JOINT snapshot: every member's weights
at that one era
* when the run gives up -> one shared plateau ladder
* whether it may deploy -> family-wise gate on the vote
WHY THE JOINT CHECKPOINT IS THE POINT: per-member selection picks each
net's own best era, and those eras differ. The resulting quartet was
never measured together at any instant, so the vote it casts live is a
configuration no OOS number ever described. Capturing all four at the
era whose vote won makes the deployed ensemble exactly the measured one.
Correct because of the era barrier (
|
||
|
AnimateDread
|
b77e7b4766 |
fix(ensemble): responsive panel + synchronized eras + combined-vote accuracy
Four user-reported/requested items, one root cause chain:
1) DEAD CONTROL PANEL in AI_HYBRID mode. All members posted custom event
id 1 and handled id 1001, and CExpertCustom broadcasts every chart
event to every filter - so each posted event ran a train chunk in ALL
N members (N*N chunks per round) and the chart thread never idled
long enough to deliver clicks/drags. profiling.csv: 99.45% of time in
OnChartEventHandler. Fix: per-instance study-event ids
(STUDY_EVENT_ID_BASE + construction order, offset above the Controls
library's ON_* codes - id 1 was also ON_DBL_CLICK, so panel
double-clicks fired training chunks). ArmStudyEvent() is the single
post site; lost-event watchdog replaces the accidental
sibling-clears-my-flag rescue.
2) WARM-UP DUPLICATION. The auto-tune sweep is deterministic over
identical features/labels, and it ends in the full MI diagnostic
suite, which the MI-share gate never intercepted on the sweep path -
four members ran four identical ~36s sweep+report blocks. First
member publishes outcome (g_ensembleChartTuneDone/Installed/Settings);
the rest apply it and skip both.
3) DEINIT STRANDED PANEL+ARROWS (user repro 18:52). Root cause from the
log: the 4,500ms budget runs from MetaTrader's stop REQUEST - a heavy
autosave in flight ate it, OnDeinit got ~430ms and died in the first
member's arrow persist ("Abnormal termination" 432ms in). Fix: early
visible-UI sweep (native prefix deletes for status/panel/dialog)
right after ClearStatusLabel, and a fast path for still-training
models - their arrows are re-rendered every era, so they get one bulk
purge instead of scan+atomic-write in the death window.
4) ENSEMBLE FEATURES (user requests): era BARRIER - members advance era
by era together; a member ahead of the slowest still-training member
declines Train() calls and its chunk budget is donated
(TRAIN_TIME_BUDGET_MS = 120/activeTrainers, UI headroom constant).
COMBINED-VOTE OOS SCORE - each member's pass-3 scan contributes its
adjusted per-bar decision (0.0 on abstain) to a shared row buffer;
the last member to finish the era scores the averaged vote vs the
mirrored Min_Vote_Open against the same target-before-stop outcomes
members grade themselves on, publishing an "Ensemble vote" line on
the aggregated panel. Member headlines now carry their lifetime win
rate with break-even.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
7dff532c70 |
fix(ensemble): shared MI diagnostics + divided chunk budget - warm-up and panel responsiveness
Two user-reported ensemble regressions, one cause each: - "getting ready is very long": every member ran the full MI diagnostic suite (headline MI, positive control, alignment, lag profile, geometry scan + winner test - ~200 permuted draws per line) on IDENTICAL features and labels, reporting the same numbers four times. First member runs it, the rest adopt with one log line. Documented caveat: if the geometry scan ever ADOPTS a winner under its gate (it never has), the adoption becomes donor-only and the gate must be revisited. - "panel not responsive": four members chunks queue back-to-back on the one chart thread - 4 x 120ms = 480ms worst-case click latency, the exact regime the 200ms note in Training.mqh already documents as broken. Ensemble members now use a 30ms chunk budget, restoring solo UI latency at slightly higher dispatch overhead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
0adaea48b6 |
fix(resume): model reload stalled training - three hardenings on the resume path
A resumed META model hot-looped pass 1 (0->100% scan oscillation, silent for
3 minutes until the stall reporter fired) because EVERY window failed at the
first AD/Wyckoff feature: the init-time param adoption called
ReInitADIndicators unconditionally, destroying five freshly-calculating
indicator instances to recreate them with BYTE-IDENTICAL params (verified by
parsing the .nnw header - the MI tuner had kept the configured settings), at
process start, on a box with 1 GB free of 31. The replacements sat cold for
6+ minutes while full-history resweeps starved the indicator threads harder.
- AdoptIndicatorParams: installs a loaded param set into the tuner and
rebuilds handles ONLY when the set actually differs from what the live
indicators run. Both call sites (resume init + panel reload) use it.
- Resumed models get the same 3 warm-up passes as fresh ones. The skip was
the shared root cause of the cold-ATR (
|
||
|
AnimateDread
|
df48c37f65 |
feat: per-family x per-side OOS breakdown in the META era report
350-era S2 verdict on SP500 H1: the meta head carries REAL ranking skill (+1.0-1.3pp mean over base, 101/350 eras clear their own 2-sigma bar, traded subset wins 66.1% at <30% coverage vs 64.5% base) but 0/350 eras produced a positive cov x (p - BE): the candidate stream sits 3pp under the derived geometry's 67.5% break-even and ~2.6pp of recovered skill cannot bridge it. Skill plateaued by mid-run (1.28pp -> 1.05pp), so more eras only buy multiplicity, and the deploy gate correctly shipped nothing. The aggregate can hide a deployable subset (one family/side clearing BE blended with junk), so the META era line now decomposes the SAME traded population into MA/RSI/MACD/Ichimoku x LONG/SHORT cells, each as traded/candidates base->traded win rate. 32 cells is a best-of-N search by construction - any candidate cell faces the family-wise rule before belief. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
444909d0a3 |
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus
The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
0848c8a16c |
fix: live inference queried the 1-tick forming bar - a window training never built
RefreshLatestSignal ran at the first tick after a bar opens and built its window at r=0: series index 0 at that instant is a candle with one tick of data - (close-open)/atr ~ 0, high ~ low, degenerate volume, indicators on a 1-tick bar. Training never produces such a window (every labeled bar is fully closed, entry at that bar's CLOSE), so the deployed model's final timestep - the one the LSTM/HYBRID output is keyed to - was out-of-distribution on every live decision, and pass 3's deploy-gate OOS scores measured a different query than live executed. The parity index is r=1: the newest CLOSED bar, whose close IS the current price - the exact instant the label's hypothetical entry happens. Single backtests shared the old skew (same r=0), which is why the tester agreed with live while both disagreed with training. Bookkeeping split that the index change forces: m_lastBarTime/dtStudied stay anchored to the FORMING bar's open (they gate against SERIES_LASTBAR_DATE; anchoring at bar 1 would re-fire the refresh every tick), while bt - the arrow, its High/Low placement, and NMS declustering - anchors to the decision bar, now matching the rescan path's convention. Also: a failed refresh no longer trades the previous bar's signal for the whole bar. RefreshLatestSignal returns success, zeroes dPrevSignal on failure (no opinion beats a stale one), and RefreshConvergedSignal advances dtStudied only on success so the next tick retries - the tester path (m_lastBarTime) already worked this way; this is the live path catching up. FORCES RE-VALIDATION of deployed models: the effective live query distribution changes. Bundled with the backprop transpose fix's retrain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
c5acc5a7a8 |
perf: pass 1 forward-passed ~40% of bars that a later pass redid anyway
Pass 1 already skipped its feedForward on QUEUED bars, because pass 2 redoes them. The same argument covers two more bands it was still forwarding: OOS window (30% of bars) - pass 3 re-forwards every one of them calibration band (~10% of bars) - pass 2.5 re-forwards every one of them All three passes derive their bounds from the same helpers and apply the identical eligibility test, so the bar sets are equal by construction, not by coincidence. Only the two purge bands and the ineligible edge bars are visited in pass 1 and nowhere else - those keep their forward pass. The scan's copy was never the one that survived. Its arrow-cache write was overwritten by pass 3's (with the thresholded, post-training decision), its status-label paint was transient, and its predicted-class tally measured last era's weights. Those tallies move to pass 2.5 and pass 3, on the raw argmax exactly as pass 1 and pass 2 count it, so the population behind the panel's "Predicted -> Buy/Sell/Neutral" line is unchanged and stays comparable with the "Actual" line beside it, which pass 1 still accumulates over every labelled bar. Verified unaffected by the cut: dPrevSignal and m_lastBarTime are both written last by bars 0/1, which are label-ineligible and therefore still forwarded, so FinalizeTrainRun's `dtStudied = m_lastBarTime` and Lifecycle's newBarPending sentinel read the same values as before. Correctness, not just speed: batch norm is UNFROZEN during pass 1 (passes 2.5 and 3 freeze it deliberately), so every scan-time forward on a held-out bar was advancing the BN running mean/variance from data the model is graded on. Those running statistics are inference-time model state. It is the mild, unsupervised kind of leakage - feature statistics, not labels - but it fed the weights pass 3 then scored, and it is now gone. Cost: ~40% of all bars lose one forward pass per era, ~16% of net time once pass 2's backward pass is weighted in. Per-dispatch, so it lands on every backend. Both variants compile 0 errors / 0 warnings. Build tag scan-nofwd-v5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
e2c959331f |
perf: the excursion head cost 3.6x era time - cut its dispatches ~250x
Measured on exc-race-v3: LSTM era 300s -> 1087s (net 272->748s, "other" 30->337s). My estimate had been "single-digit percent". The cost is per-DISPATCH, not per-FLOP, and therefore hits EVERY backend: the head is 19k weights and ~2.4 GFLOP an era - seconds of arithmetic - but ~48k forward/backward calls x several layer submits each, and its 760-wide layer exceeds the CPU DLL's inline threshold so each one pays a real handoff. The classifier's own net time tripled too, from contention with a second pool on an already-full box. Three changes, all backend-neutral because they remove submits rather than tune threads: SCORE ONLY DISJOINT WINDOWS (~64x). Adjacent bars share all but one bar of their horizon, so 16k consecutive bars were always ~250 independent observations - the full-sample tally was never worth more than the disjoint one, it just quoted an n that was ~64x too large. Dropping it costs nothing statistically and removes 63 of every 64 forward passes. The two parallel tallies collapse into one, which is also less code. The trailing ring still advances on every bar: it needs the outcome SEQUENCE, and that is array lookups, not a forward pass. TRAIN ON EVERY 4th PRIMARY BAR (4x). The target is low-dimensional and strongly autocorrelated - neighbouring bars carry near-identical excursion information - so per-bar training buys resolution the target does not have. Strided on ATTEMPTS, not acceptances, so a stretch of unlabelled bars cannot silently change the spacing. OWN TIMING COLUMN. The head's passes were landing in the era line's "other" bucket, which is how a 3.6x regression read as an unexplained jump in the one column nobody attributes. A cost that cannot be seen in the timing line cannot be traded off against anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
950b0fdab0 |
diag: name the cause when every feature window fails, and enforce the width contract
Era 0 stalls with "NOT ONE of 54681 scanned bars produced a usable feature window, windows ok=0 failed=54681" and nothing else. That line reads identically for a cold ATR, a conditionally-missing optional feature block and an out-of-range index, so it cannot be diagnosed without one restart per hypothesis. Two changes: 1. WIDTH CONTRACT in BufferTempData. Every enabled block must emit exactly m_neuronsCount values on EVERY bar. A block that emits its values on some bars and skips them on others (indicator, panel or series unavailable for that bar) does not merely shorten the window - it SHIFTS every feature after it into the wrong slot, and the net then trains on silently misaligned inputs that still look like a valid window to everything downstream. Now rejected, rolled back and reported once, naming the optional blocks (XA / SPR / swing context) as the ones carrying an availability test. Worth having independently of the current stall. 2. BuildFeatureWindow records WHICH lookback slot rejected and how much of the window was assembled, and the pass-1 stall report renders it: "slot 0 of 20 REJECTED (window had 0 of 760)" is an indicator warm-up or history-edge read; "every lookback bar ACCEPTED and the window was still short: 640 of 760" is a missing 6-value block. No behaviour change on a healthy run: the width check is an equality that already holds, and the diagnostics render only inside the total-failure branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
2d28f6542b |
feat: excursion-size head (Stage 1, measurement only)
Direction is closed - normalised asymmetry fails on three instruments with a working positive control, and the classifier's own best-of-999 era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is a different question and RANGE clears at ~4x its null. Checked the denomination before building on that, since the source memo warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is predictable" is a claim about travel RELATIVE to current ATR, not a restatement of "ATR is autocorrelated". It is exactly the part a fixed multiple (stop 3.31*ATR, target 1.64*ATR) discards. A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches ladder rung k) upward and downward. Survival parameterisation rather than regressing the multiple, because it needs nothing new from CNet: sigmoid outputs and the per-neuron delta the `total != 3` branch already applies (a quantile head would need a linear activation and a pinball gradient in Network.mqh, Network.cl and the DirectML path, on a class four topologies share). Targets are free - m_ladderUpAt already records first-touch age per rung with 0 meaning never reached. Separate net, not extra outputs on the classifier: more outputs would change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push the count off 3 - the exact condition backProp uses to select the joint softmax gradient the 3-class head depends on. The classifier is bit-for-bit unaffected and this is removable without trace. STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the constant per-rung base rate - the baseline a fixed ATR multiple already assumes - with both predictors fitted IS and evaluated OOS, so neither gets a look at the test set. Positive skill justifies Stage 2 (drive SL/TP and sizing off ExcursionQuantile, which is defined and deliberately uncalled). Zero or negative means ATR already carries everything and Stage 2 must not be built. Trains only on primary occurrences: the replay queue oversamples for CLASS balance, and a direction-balanced sample is a biased SIZE sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
0c8b4dc30d |
fix: the deploy gate graded the un-thresholded model
coveragePct, dirPrecPct and the declustered TRADED tally were all computed from oPrevSignal - the RAW argmax - while the live order, the arrow and the panel all run on oDeploySignal, which is argmax AFTER the confidence threshold. The gate was certifying a strategy the EA does not trade. Invisible until now: the threshold sat at ~0.02, so the two populations were the same set. The held-out calibration slice ( |
||
|
AnimateDread
|
2189316c35 |
fix: the operating point was fitted on bars the net had memorized
FitDirConfThreshold harvested its margin histogram from pass 2's own
backprop samples. Pairing every fit against the same era's OOS result
shows what that measured:
PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp
PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp
LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp
The gap grows monotonically while OOS stays flat, so within a handful of
eras the curve stops describing behaviour on unseen bars. That is fatal
here specifically, because the objective branches on the SIGN of
(p - break-even): the memorized curve reads +12pp at 95% coverage, so
coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire
on every bar. The "p < p0 -> get more selective" branch, which is the
actual regime and the entire point of
|
||
|
AnimateDread
|
d919a4aea2 |
feat: 10-bar decluster window + alternation on every signal consumer
SignalClusterWindow 3 -> 10 for all topologies. On H1 a 3-bar window
collapsed only the tightest runs and left visible clusters at every
turn; 10 bars is closer to the spacing of genuinely distinct setups.
ALTERNATION. Rule 1 only collapses a same-direction run INSIDE the
window; past it a second Buy is emitted with no Sell between, giving
Buy/Buy/Buy/Sell. With both directions tradeable that sequence is the
model re-entering a move it is already in rather than finding a new
one. The kept sequence must now alternate: the first signal passes,
and after that a direction passes only if the last KEPT signal was the
opposite one.
Added to ALL THREE consumers, with identical logic, because they must
agree:
- NmsLiveAccept -> the live trade
- pass 3's OOS replay -> the tally the deploy gate grades
- PruneDirectionalClusters -> the drawn history
A rule applied to only some of these certifies one strategy and trades
another - the same defect class as the geometry the gate certified
while OpenParams placed something else (
|
||
|
AnimateDread
|
ccfbc62561 |
fix: the recall gate was unsatisfiable and the LR decay was a spiral
Both made the run structurally unable to succeed, independently of any
signal in the data. Found by reading the 13:01 log.
RECALL GATE. m_objectiveMet required Buy, Sell AND Neutral OOS recall
each >= 40%. First-touch resolution (
|
||
|
AnimateDread
|
ba13eefecc |
fix: a resumed model cached a cold ATR as permanent, so it never trained
BufferTempData cached EVERY failure - m_featureCacheHasValue[idx]=true with m_featureCacheValid[idx]=false - and the cache never re-tries a miss. So a single feature read taken before the terminal had finished calculating the indicator buffers marked those bars unusable for the rest of the process, even though the data arrived milliseconds later. MT5 fills an indicator's buffers asynchronously after the handle is created, and a cold ATR returns 0 for EVERY index, not just its warm-up tail. BufferTempDataCompute rejects a bar with no ATR (correctly - the price features would be meaningless), so the whole window failed, and the whole cache was poisoned. Only resumed models were hit, because only they read features that early. Topology.mqh sets m_warmupPassesRemaining = netLoaded ? 0 : 3: a fresh start sits through three separately-scheduled Train() calls before anything touches a feature, which is exactly what those passes are for. A resumed one skips them and TuneIndicatorsAndTrain drives StartLabelCachePrebuild and the MI report from the first chart event. Its rationale - "a restart already has a proven-synced history" - holds for HISTORY and not for INDICATORS, which are recreated every process start. Downstream: BuildFeatureWindow failed on every bar of every era, so add_loop never went true, so pass 2, pass 3, the era counter and the checkpoint were all skipped and pass 1 swept 0->100% forever. The "0 samples" MI report line at startup was the same failure, four seconds earlier, already visible in the log. - a miss is now cached only when it is PERMANENT; the two "not ready yet" guards mark m_featureFailTransient and are recomputed on the next visit. Steady-state cost is ~ind_Periods bars per era, not 54k. - an era that discards itself now drops the feature cache before restarting, so any remaining cause of this state self-heals instead of looping. Deleting the .nnw "fixed" this only by turning the model back into a fresh one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
464a0fe19d |
diag: an era that discards itself now says so instead of scanning forever
add_loop is exactly "at least one bar produced a usable feature
window". When it stays false, pass 2, pass 3, the era counter, the
checkpoint and every log line in the era-end block are ALL skipped:
Train() returns having done nothing, m_eraResumePending is still false,
and the next call restarts the SAME era from bar 0. That is an
infinite 0->100% "scan" loop that prints absolutely nothing - the only
remaining silent restart path in Train(), and it matches the reported
symptom exactly.
Pass 1 now counts usable vs unusable windows and reports at the pass
boundary, which demonstrably executes:
- total failure routes through ReportTrainStall (already capped at
one line a minute, and carries the run-state flags) naming the
counts, the required window width and the bar count
- success prints how long the scan took and how many samples it
handed to pass 2, but only once the era has passed 10s - a fast
era stays as quiet as before, a slow one distinguishes "advancing"
from "sweeping the same bars forever"
A PARTIAL failure is normal and deliberately does not shout: pass 1
walks oldest-to-newest and the deepest bars predate the indicators'
warm-up, so those windows fail and are cached as misses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
AnimateDread
|
1a5157befc |
fix: training could only advance one 120ms chunk per bar
ScheduleTrainingIfNeeded() armed the next Train() call only when dtStudied < lastBarDate. That watermark test is right for a CONVERGED model - one inference refresh per new bar - and wrong for a training run, because Train() is chunked: it does ~120ms of work and yields, needing thousands of calls to finish one era, and every one of those calls has to be armed from there. dtStudied is two incompatible things. Train() sets it to the training WINDOW START (~2008); FinalizeTrainRun() sets it to the last bar SCANNED (~now). So the moment any run finalized, the scheduler went silent until the next candle closed. On H1 that is one chunk per hour. The symptom was indistinguishable from a hang: no era lines, no heartbeats, not one of the six instrumented stall branches - because Train() was not being CALLED. The TRAIN STALL line that caught it reported runActive=Y only because m_trainRunActive had been set microseconds earlier in that same call, and eraResume=N proved no era was in flight. Two log bursts, 28 minutes apart, exactly one H1 bar. Before |
||
|
AnimateDread
|
3855d4666a |
diag: the heartbeat could be outrun by the condition it watched for
It fired only on 4096-item boundaries once an era had already run 60s. Those boundaries are all crossed in the first few chunks of pass 1, so an era that became slow AFTER them printed nothing at all - which is precisely what happened: 20 minutes, four pegged cores, zero heartbeats. I read that silence as "the era loop is never reached" and went looking for a wedge above it. The silence may simply have meant "past the last boundary". A diagnostic whose trigger can be outrun by the condition it watches for is worse than no diagnostic, because it produces confident wrong conclusions. Now time-gated: checked every 256 items (the mask only keeps GetTickCount off the hot path), prints when the era has run >60s and >30s since the last line, up to 12 per era. Progress/phase for the panel is still published on every call, before any gate. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
b461844767 |
fix: prebuild and era sized different windows; diag: Train() names its branch
TWO things, one incident. 1) THE BUG I SHIPPED IN |
||
|
AnimateDread
|
783fd9e7a6 |
fix: the panel showed "100%" for the whole of pass 1
The simple panel derived its percentage from pass 2's counters: (m_isTrainCursor+1) / max(m_isTrainQueueCount,1). During pass 1 those are 0 and 0, so the expression is (0+1)/max(0,1) = 100%. An era spends its first pass scanning ~38k bars - minutes of work - and the panel reported that phase as finished the entire time. Observed by the user as "started learning at 100% of their era and are stuck there", and it actively misled the diagnosis: the one number on screen said the opposite of what was happening. The UI cannot fix this on its own - it can see pass 2's counters but has no way to know which pass owns them. So each pass now PUBLISHES its own progress and a short phase name through TrainHeartbeat (which every pass already calls per item), and the panel just displays them: "learning (era 45, scan 34%)". Published before the heartbeat's 4096-item journal gate, so the panel updates continuously while the journal stays quiet. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
694b75686e |
diag: slow eras must explain themselves - heartbeat + era time split + pass-1 paint
The 23:42 restart left all four charts grinding ~25x slower than the 18:01 baseline (era lines in 86 seconds there; 20+ minutes of nothing here), and NOTHING could say why from outside: pass 1 logs nothing, its status paint sat inside the !wouldQueue branch so the IS sweep - 80% of the pass, processed FIRST - painted nothing either, the VPS has no debugger for a thread stack, and the hourly new-bar cache invalidation cancels and restarts an unfinished era, so a slow era can stay invisible FOREVER. Externals gave: four chart threads at ~95% pure user-mode compute, DLL pool idle, no file writes. That narrows it to "MQL5-side per-item work in the era passes" and no further. So training now explains itself: - TrainHeartbeat: one line per 4096 processed items, only after an era has already run 60s, at most 6 lines per era - a healthy era stays exactly as quiet as before. Reports position and the cumulative split: feature-window builds vs net forward/backprop vs everything else. Hooked into all three passes. - The era summary line gains "| ERA TOOK Ns (feature windows X, net fwd/back Y, other Z)" whenever an era exceeded 120s. - Pass 1 paints its progress for QUEUED bars too, not just the OOS slice, so the panel shows "learning (era N)" instead of sitting on the idle writer's "Getting ready..." for the entire IS sweep. The label is throttled internally; painting per bar costs nothing. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
0c85c54a5b |
fix: a restart no longer loses the measured geometry or the training window
Terminal restart, 22:25: all four resumed models sat on empty windows with enum 2:6 barriers. Three interlocking causes, all visible in one log excerpt: 1) THE PRE-SCAN WINDOW WAS SIZED BY THE SAVED WATERMARK. A resumed model's dtStudied sits at its last studied bar, so Bars(dtStudied, now) ~ 0 and the resumed-model MI pre-scan built a zero-bar "complete" label cache - logged as "Buy: 0 | Sell: 0 | Neutral: 0". Train()'s own era start RESETS dtStudied to the training-window rule before computing its window; the pre-scan did not. The rule is now factored into TrainWindowStart() and both use it. The scan also refuses to arm before SERIES_SYNCHRONIZED (it ran in the same second as OnInit), and deployed models keep their watermark - for them it gates inference recency, not a training window. 2) THE HORIZON LATCHED ON AN INDICATOR WARM-UP. ComputeBarrierHorizonBars ran against a ZigZag with 0 calculated legs, fell back, and EnsureBarrierHorizon latched fallback(32) x slMult x tpMult = 384 for the process lifetime. A leg-starved horizon is now PROVISIONAL: re-resolved on the next rebuild, the label cache wiped if it moved (labels from two horizons answer different questions), and the geometry deriver refuses to run from it - a pair derived over a warm-up window would get PINNED. 3) THE DERIVED GEOMETRY WAS NEVER PERSISTED. The .cfg is written at model creation and at weights-reset - both BEFORE era 0 derives - so the measured pair lived only in memory: every restart read back zeros, adopted nothing, fell back to the enum barriers, and the era-0-only gate meant a resumed model could NEVER re-derive. A full day of training on 3.33/1.62 resumed as 2:6. Now: the settled pair is pinned to the .cfg the moment derivation completes (one-shot, atomic write), and the derive gate accepts any model with no pinned pair, not just era 0 - mid-run stability is carried by m_geometryDerived itself, which never allows a second derivation. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
199726f651 |
fix: a one-sided era can no longer become the best checkpoint
Measured on HYBRID, era 29 of the first win-scored run: the model collapsed
to always-Buy and was crowned "new best selection score 67.1%". Under
win-based scoring that is not a coincidence - the always-call-the-drift-side
model IS the chance reference, so it scores exactly chance (P(winLong) ~ 67%
on SP500), while every honest two-sided era scores 63-66% because shorts win
less often against the drift. Raw score ranking therefore actively prefers
the degenerate model, every regression restores back to it, and live NMS
collapses its near-constant signal to ~25 trades per era - observed as
"hybrid barely trades".
bothSidesLive already blocked one-sided eras from DEPLOYING (tradeableOK,
|
||
|
AnimateDread
|
5cef0947f4 |
fix: the deploy gate was benchmarking a win rate against a label frequency
The gate rests on an invariant stated at ExpertSignalAIBase.mqh:199 - under a
driftless walk P(touch +k before -m) is m/(m+k), and break-even for a k:m trade
is ALSO m/(m+k), so "beats chance" and "is profitable" are the same test.
That invariant needs reward >= risk, and the measured geometry no longer
satisfies it. With target 1.62*ATR and stop 3.33*ATR, break-even is 67.3%, but
both-won bars were stripped out of Buy and Sell so the label base rate read
37.5%. chancePrecPct is max(BuyTotal,SellTotal)/bars, so the gate was clearing
models nearly 30pp short of break-even: 42% "directional precision" is +4 sigma
against 37.5% and loses money on every single trade. Live since
|
||
|
AnimateDread
|
19dfb91108 |
feat: fitted directional confidence threshold - selectivity gets a mechanism
The training loss and the selection metric wanted different things and only the second one knew it. Logit-adjusted cross-entropy has no term for "how often should I trade", so the head calls a direction on 87-91% of bars. The selection metric is precision x coverage credit, saturating at the coverage floor - above the floor extra calls earn NOTHING and only precision counts. So selection wanted few good calls, the loss produced many mediocre ones, and all selection could do was pick the least-bad era out of what it was handed. Nothing pushed the model toward selectivity. This gives the decision RULE the policy instead of distorting the loss (which is estimating class probabilities correctly, and a probability estimate should not be bent to encode a trading policy - Elkan 2001: estimate, then choose the operating point separately). AdjustedSignalFromSoftmax now abstains unless the winning direction's softmax margin over its best rival clears a fitted threshold. Margin, not the winning probability: the latter moves with overall calibration rather than with how close the decision actually was. Fitted on IS, applied to OOS and live. Pass 2 already forward-passes every IS sample, so the margin histogram is harvested there for free (primary occurrences only, so the oversampled replay queue cannot skew the operating point); the fit runs at the end of pass 2, BEFORE pass 3, so the deploy gate grades the thresholded model on bars the threshold never saw. Fitting on pass 3's own predictions would be choosing the operating point on the data being graded - the best-of-N error corrected in five other places here. Objective: maximise IS directional precision subject to still clearing the SAME coverage floor the deploy gate uses (base rate x 0.25, re-derived locally so the two cannot drift apart). Swept top-down in one pass; ties go to the LOWER threshold, since equal precision for less coverage is strictly worse. Under DIR_CONF_MIN_FIT_CALLS (200) it runs unthresholded rather than on a guess. The threshold is part of the MODEL, not the run: captured with Net.CaptureWeights(), restored with the weights at both restore sites, and appended to the .cfg under the same length-guard convention so a deployed model reloads at the operating point its gate actually cleared. A pre-2026-08-09 .cfg reads 0.0, which is exactly the behaviour it was trained under. Per-era line now prints "@margin>=X.XX" next to coverage, so a coverage drop can be attributed to the operating point rather than guessed at. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
371f8aaecd |
fix: the Adam second moment was never Adam - all four tiers
Root cause of the B=32 regression, and it predates F4 entirely. Every Adam
kernel stored v already square-rooted and then fed that stored value back in
as if it were the variance:
v_new = sqrt(b2 * v_old + (1 - b2) * g^2)
That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below
unit scale, so the denominator stops tracking the gradient and Adam degrades
into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll
(batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a
constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where
a scale-invariant optimizer gives the same distance for both. After the fix
all six magnitudes read 1.199 and v tracks |g| exactly.
It hit conv/LSTM specifically because they sit behind a batch-norm with
running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep
in the degraded regime - while the dense stack near the loss stayed in the
working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/
0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already
squared v back for gamma/beta and its comment named the kernels as wrong,
which is exactly why gamma/beta kept training while the stages behind froze.
Persisted .nnw needs no migration - v keeps its std-dev meaning.
Also, the two ways F4 exposed it, both mine:
- No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods
(Krizhevsky 2014; Granziol et al. 2022), applied once in
InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD.
- Plateau patience denominated in eras, so raising B made the ladder 32x more
impatient in its only unit. PAI converged at era 41 on ~49k updates where
the same config had been finding new bests at era 1028.
TrainPlateauPatienceEras() stretches it by the same sqrt(B).
TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23
eras per stage, not 8 -> 45). Both helpers are identities at B=1.
Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The
perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy
alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into
the ranking key rather than checked at deploy time so a one-sided era cannot
become best-so-far in the first place.
Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded
CAppDialog teardown that sat ahead of it - the same ordering inversion the
rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit
(vs ~1.1 s for the three that finished) having reached none of its cleanup,
so its arrows stayed on the chart. Steps are now timed in the log.
PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot
as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now
walks every object type and reports the object counts when both are zero.
Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt.
FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
0c01dc279b |
feat: mini-batch gradient accumulation (F4), front-end-aware capacity budget (F6), split Wyckoff categoricals (N1)
Completes the 2026-08-09 training audit. FORCES A RETRAIN of every
Wyckoff-enabled config (N1 re-keys the fingerprint), and BOTH DLLs must be
redeployed alongside the .ex5 - they carry new exports.
F4 - mini-batch accumulation, TRAIN_BATCH_SIZE=32. Training was pure online
SGD (one weight update per bar), which is the mechanical source of the
era-to-era whipsaw every downstream guard was built to cope with. The O(n^2)
outer product is native - AccumulateWeightGrad / AccumulateWeightGradConv /
AccumulateBufferInto in Network.cl, WarriorCPU and WarriorDML - while the
optimizer step is host-side MQL5 shared by all tiers (ApplyAccumToBlock), so
there is one Adam/SGD implementation instead of four that can drift.
- the LSTM needs no outer-product kernel (WeightsGradient already holds the
sample's full dW) but could NOT simply be left un-zeroed between samples:
CPU_LSTMSeqBackward/DML_LSTMSeqBackward memset it on entry. Hence a
separate accumulator plus an elementwise add.
- batch-norm gamma/beta accumulate in host arrays, not new BatchOptions
slots - BN_OPT_STRIDE is baked into every persisted .nnw.
- scoped to pass 2; online learning keeps immediate updates. Every save /
checkpoint / scoring boundary flushes, scaling by the real sample count.
- degrades to per-sample updates (one log line) on a tier that cannot
accumulate, so old devices and DLL-free builds are unaffected.
- verified offline: DirectML/batch_accum_check.cpp drives the real exports
against an independent reference; at B=1 the accumulator matches the
shipped unbatched kernel's own gradient to 1.1e-16. Math only - the
in-situ check remains the per-layer dW/W report on a real era.
F6 - ComputeFirstLayerWidth budgeted against the RAW input width even where a
conv/LSTM front end had already reduced it, so an LSTM's dense stack was
charged for 1,280 inputs when it receives 64. Confirmed from the deployed
.cfg files: CONV, LSTM and HYBRID were all pinned at the 16-unit floor. Now
budgeted against the front-end output and capped at it (never fan out), with
the derivation reordered so both stages settle first.
N1 - EventCode/EventPhase/StructuralPhase are signed categoricals packing
direction and Wyckoff stage into one scalar across a sign discontinuity. Split
into direction + [0,1] magnitude, the same convention the base OHLC block uses.
Information-preserving; 13 readings now occupy 16 inputs.
Compiled clean (0 errors, 0 warnings); both DLLs rebuilt.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
274630f802 |
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric
Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |