forked from mnbvc188199/Warrior_EA
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
AnimateDread
|
778b6c09c6 |
feat(gates): derive the recall floor instead of configuring it, publish what the configuration can PROVE, and stop an arbitrary member driving live exits
Three changes, all from the same principle: measure what is there before aiming
at it, and never certify a number you do not trade.
1. THE RECALL FLOOR IS DERIVED, AND IT MOVES BELOW CHANCE.
MinRecall=40 was a constant doing a statistical job. Its reference point is the
33.3% recall a zero-skill 3-class model gets on EVERY class, and against that the
constant was accidentally calibrated for exactly one sample size: on USDJPY CONV
(n_eff 195) 40% is chance + 2.0 SE; on SP500 PAI (n_eff 42) the same 40% is chance
+ 0.9 SE. One chart was being held to a bar twice as strict as the other, for no
reason anyone chose.
CollapseRecallFloorPct() computes it per class from that class's own effective
sample - EffectiveSampleSize(), so the overlap deflation the rest of the gates use
applies here too - as chance - EDGE_MIN_SIGMAS x SE. 26.5% at n_eff 195, 18.8% at
n_eff 42.
BELOW chance, deliberately, and this is the substantive change rather than the
arithmetic. This gate's only job is refusing to call a COLLAPSED model converged.
It is not a quality bar; the deploy gate is the quality bar and it is already
rigorous (chance + 2 SE on the deflated sample, Sidak over candidate eras, then the
cross-instrument pooled certificate). A convergence gate that ALSO demands
provably-above-chance recall on all three classes double-counts that job, and it
has failed that way twice here: MinRecall=60 blocked every SP500 H1 run in 2026-07,
and the 40 that replaced it made Neutral structurally unreachable once first-touch
resolution cut Neutral to a 0.65% residue. A floor nothing can reach does not make
a funded account safer, it stops the run converging at all.
Testing significantly BELOW chance instead catches what a fixed 40 was actually
catching - a model that has stopped emitting a class - and cannot become
unreachable by construction. It also fixes the direction the old constant scaled:
it now widens on a thin OOS window, where low recall genuinely cannot be told from
noise, and tightens on a rich one. Today's SP500 PAI (Buy 51 / Sell 18 / Neutral 30)
is still correctly blocked on Sell.
This also resolves a standing contradiction the code half-admitted at the
isBetterEra comment: selection ranks on coverage-weighted PRECISION while
convergence gated on RECALL, so a sparse high-precision abstainer - precisely the
model that could clear the deploy bar - was blocked by the floor.
The era line now PRINTS the derived floor. Anyone comparing these recalls against a
remembered "40" is reading the wrong bar.
2. DETECTABILITY: WHAT THIS CONFIGURATION COULD PROVE, BEFORE IT TRAINS.
The DEPLOY BAR line states the bar. It never said what reaching it would take, and
that is the actionable direction. ReportDetectability() inverts the same identity -
the gate passes when edge >= z x sqrt(p(1-p)/n_eff), so certifying an edge d needs
n_eff >= z^2 p(1-p)/d^2 independent calls, hence L times as many raw ones - and
prints a +2 / +5 / +10pp ladder as required independent calls, raw calls, and share
of the OOS window, marking any rung that needs more than the window holds IMPOSSIBLE.
Every term is a property of the CONFIGURATION - geometry via break-even, horizon via
mean label lifespan, window via oosCutoff - so no amount of training moves any of
them. It fires once, at the first healthy sweep, beside ReportFeatureHealth, for the
same reason: that is the first moment the bar grid, the measured geometry and the
lifespan are real numbers rather than defaults. It gates nothing.
This is the EdgeFinder discipline applied to our own gate: establish what the market
and the measurement design have to offer, then point the net at it - rather than
spending a thousand eras chasing something this OOS window could never certify.
3. AN ARBITRARY MEMBER WAS DRIVING LIVE EXITS AND TRAILING (user-identified).
Every ensemble member ran
g_LiveAISignedConfidence = SignedAIConfidence();
unconditionally, every tick. Last writer wins. Its consumers are the AI early-exit
route (CExpertSignalCustom::LiveSignedConfidence) and TrailingIntelligent - so on a
four-model chart an LSTM entry could be closed, and its stop moved, on the
Perceptron's opinion alone, decided by scheduling order. Not the vote, not a
weighted blend.
Now the mean across registered members, matching how the ensemble actually trades:
the open decision is the weighted-average vote, and an abstaining member contributes
0 and dilutes exactly as it does there. Members still training read 0, so a
half-trained ensemble reads WEAKER rather than louder - the safe direction for an
exit trigger. Deployed and paused members are included, which is the opposite of the
era barrier's exemption rule and correct for the opposite reason: that one asks who
must be waited for, this asks who has an opinion.
Latent today and staying that way for now by choice - Min_Vote_Close ships Disabled
(101, unreachable on both scales it drives) and TrailingStrategy is off, so live
exits are SL/TP only and the certified hold-to-barrier win rate is what actually
gets traded. Fixed now precisely because the plan is to enable vote exits once the
models are accurate, at which point a scheduling-order exit would be both harmful
and very hard to see.
STILL OPEN, and needs a decision before vote exits go on: the member gate and the
ensemble vote gate both grade hold-to-barrier, so enabling vote exits makes the
certified number stop describing the traded one. Warrior_EA.mq5 currently argues
barrier models may keep vote exits because "their label IS the vote's own horizon" -
that does not hold, since a vote flip at bar 5 of a 64-bar horizon is not the
target-before-stop outcome the gate measured. Either grade the OOS call on the real
exit rule (first of SL / TP / vote-flip / horizon) through the fill engine, or set
HoldToBarrier for ensemble members so the policy cannot drift from the certificate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
AnimateDread
|
fca610fea0 |
fix(indicators+panel): the dead handle is MEASURED now - recreate it; and order the ensemble panel by member, not by who published first
THE ANSWER, off the instrumentation added in |
||
|
AnimateDread
|
be396749fc |
fix(ensemble+depth): the barrier had no liveness escape, and the depth gate could not report the one state the evidence pointed at
Two charts (USDJPY 50,179 bars / XAUUSD 33,982) sat at era 0 for 38 minutes with
four of their eight members completely silent. Nothing in this commit guesses at
why the sweep fails - the last five guesses were all wrong. It makes the failure
say what it is, and stops one broken member taking its whole chart down with it.
WHAT THE LOG ACTUALLY SAYS, before any of this.
- The running build IS
|
||
|
AnimateDread
|
d9f834d01d |
fix(buffers): revert the MA +1 - it asked for a bar that does not exist and stopped every chart
REGRESSION I INTRODUCED IN
|
||
|
AnimateDread
|
45c9e211b3 |
feat(depth): prime -> settle -> sweep, and name which handle is short
"Max bars in chart" is set to Unlimited, so the static-terminal-limit reading in |
||
|
AnimateDread
|
7e63a8be01 |
fix(depth): route EVERY ResizeBuffers call site through one indicator-depth gate
|
||
|
AnimateDread
|
1dda479261 |
fix(train): clamp the sweep to indicator-servable depth - the scan wall was CopyBuffer, not a cold indicator
Symptom: on a 3-chart run with contention ruled out (SP500 sitting at era 2552),
USDJPY and XAUUSD produced 0 usable windows out of 50,163 and 33,966 - forever,
re-sweeping on every discard, which is the panel oscillating 0->100%.
Bars() is the PRICE series depth. A CUSTOM indicator's is not: MT5 calculates it
in its own context bounded by "Max bars in chart" (TERMINAL_MAXBARS), and
CopyBuffer past that limit does not short-read, it FAILS - so CDoubleBuffer keeps
nothing and EVERY index answers EMPTY_VALUE. ADMovingAverage is the only custom
indicator whose feature block REJECTS on EMPTY_VALUE (ADZigZag, also CiCustom,
neutral-fills; RSI/MACD/Ichimoku/ATR are built-ins served at any depth), so the
sweep died on feature 25 of every bar while the 24 price features under it were
fine. That is exactly the "window had 24 of 832 values" the stall report named.
Perfectly depth-correlated, measured 2026-08-17:
SP500 16,234 bars -> era 2552 XAUUSD 33,982 -> 0 windows
XTIUSD 16,611 bars -> era 71 USDJPY 50,179 -> 0 windows
This RETIRES the 2026-08-17 cold-indicator reading of the same stall.
|
||
|
AnimateDread
|
1cf4c57d57 |
fix(altdata): median-fill instead of zero-fill, and a one-shot feature-vector autopsy
ALT-DATA AUDIT. The files themselves are healthy - all six symbols, 6,073 daily
rows, 2010-01-01 to 2026-08-17, no constant or degenerate columns, sane tails
(mac_cpi/mac_unemp flat ~47d is monthly data behaving correctly). The problem is
not the data, it is what happens where the data ISN'T.
CAltDataPanel::Features() returned an all-ZERO vector for any bar older than the
file's first row, and left blank cells at 0 too. Both were deliberate ('the block
is additive context and must degrade, never reject the bar') and that reasoning
holds for the CHANGE columns - but half these features are LEVELS: vix, ivol,
mac_y10, mac_cpi, mac_unemp, eia_util. For a level, 0 is not a missing reading,
it is an impossible one far outside the series' range. VIX does not visit zero.
And the spike lands in exactly the wrong place. Every alt file starts 2010-01-01
while the charts run far deeper - USDJPY H4 reaches ~1994, roughly HALF its
history - so 'alt block is all zeros' is precisely the predicate 'this bar is
older than 2010'. The IS/OOS split is chronological, so that predicate covers
~half of IS and none of OOS: an in-sample feature guaranteed to be useless
out-of-sample, and a bimodal input for the first BatchNorm to normalise. Not a
lookahead leak - a distribution corruption, which is quieter and was never
reported anywhere.
Now filled with the column MEDIAN over the covered range. A constant cannot leak
whatever its source - it takes the same value on every pre-coverage bar, so it
carries no information about which of those bars won - which is what makes a
median computed over later data legitimate here. Median not mean because the
series are skewed. Blank cells get the same treatment (eia_stk_idx1y alone has
181 blanks in 6,073 rows) and the count is now logged at load.
THE BACKOFF WAS ALREADY THERE AND WAS DEAD. Training.mqh arms m_coldSweepTick on
m_featureFailTransient, but only the open/ATR guards ever set that flag, so
f0cf659's cold ADMovingAverage looked PERMANENT and the sweep re-ran at full
speed forever. Setting the flag in the indicator guards revives the mechanism
that was already designed for this; no second backoff was needed and the one I
first wrote has been removed in favour of it.
SELF-HEALING, as asked. ReportFeatureHealth() runs once, the first time pass 1
produces usable windows, samples ~400 bars spread across the whole training range
and names every feature slot that is CONSTANT or mostly-zero, tagging alt-block
slots as alt[i]. Both of today's failures were the same shape - a block silently
produces nothing while every downstream number stays plausible - and neither an
accuracy figure nor a model can tell 'this feature is always 0' from 'this
feature is genuinely 0 here'. Evenly spaced sampling so a block that dies only in
deep history is caught as surely as one dead everywhere. A report, not a gate:
a rare-flag feature can be legitimately constant, and refusing to train would
turn a diagnostic into an outage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
AnimateDread
|
f0cf659945 |
fix(features): a cold indicator is TRANSIENT, not a permanent miss - and back off instead of re-sweeping
Six fresh instances on USDJPY and XAUUSD swept 33,965-50,162 bars and produced
ZERO usable feature windows, repeatedly, for 40 minutes and 239 stall reports,
without ever completing era 0. The four instances already warmed up before those
charts were attached trained normally throughout.
THE STALL REPORT NAMED THE SPOT EXACTLY: 'lookback slot 0 REJECTED the bar
(window had 24 of 832 values)', and 24 is the core block to the value - 4 price +
5 swing + 4 range + 4 volume + 6 time + 1 ATR. So feature 25 was the wall, and
feature 25 is the first value of the MA block. The same 24 appeared on XAUUSD
against a 816-value window (51 features/bar vs 52), which is what ruled out any
symbol-specific data gap: the wall sits at a fixed feature index, not a date.
ADMovingAverage is a CUSTOM indicator, so MT5 fills its buffer asynchronously and
returns EMPTY_VALUE for EVERY index until it has calculated - not just the
warm-up tail. That guard did not set m_featureFailTransient, so every bar of the
sweep was cached as a PERMANENT miss. This is precisely the failure the ATR guard
twenty lines above it was fixed for on 2026-08-10; the fix was never propagated
to the indicator blocks that follow. RSI, MACD and Ichimoku had the same defect
and are fixed too. (The Donchian high/low guard is a break into a
degraded-but-usable path, not a rejection, and is deliberately left alone.)
IT ALSO SELF-SUSTAINED, which is why it never recovered. The ok=0 self-heal drops
the feature cache and re-sweeps immediately, so each stuck instance spent every
millisecond re-reading 30-50k bars - six of them at once, on a six-core box,
competing for CPU with the very indicator calculation they were all waiting on.
The recovery was preventing the recovery. A transient total failure now re-arms
m_warmupPassesRemaining, yielding the CPU for a few separately-scheduled Train()
calls - the same mechanism a fresh model already uses to let history sync finish,
pointed at indicator warm-up instead.
Verified in the terminal journal first: indicators load and unload in matched
counts and there is no OOM, so this is NOT the
|
||
|
AnimateDread
|
ee3682d949 |
fix(features): collapse only the anchor's own run - leave lagged readings put
User's call before deploy: "I would rather avoid lagging so the NN finds
accurate patterns." Correct instinct, and it picks the conservative variant.
|
||
|
AnimateDread
|
110b38470a |
fix(features): dedup the alt block BY VALUE - aba9bd2 broke D1 charts
|
||
|
AnimateDread
|
aba9bd2bea |
perf(features): the external block enters the window once, not once per bar
Measured on the live SP500 D1 export (6073 rows, 13 features, 5888 simulated 16-bar windows): distinct values per feature per window : 1.7 - 2.7 of 16 slots variance in the first 13 PCs : 96.5 - 97.0% components for 95% / 99% : 12 / 17-19 effective rank (entropy) : ~11.5 208 inputs carrying about 12 dimensions. Only 6 of the 13 features move daily (VIX complex, USD, the rates trio); 5 are weekly (COT, EIA, output gap) and 2 monthly (CPI, unemployment). The lookup is as-of by bar open time into a DAILY file, so bars sharing a calendar day are byte-identical by construction. The cost is NOT overfitting capacity - collinear copies span ~12 directions, not 208, so an earlier claim that this wasted 26% of the model overstated it. It is GRADIENT WEIGHTING. Batch norm standardizes each of the 208 coordinates independently; that rescales the copies without decorrelating them, so one factor arrives on 16 unit-variance coordinates, each weight takes a full-size step, and the factor's aggregate coefficient moves ~16x faster than a per-bar price feature's. The network was biased toward the external block by a factor of the window length - and pointing the wrong way, since these features cleared only a marginal incremental screen while price is the base signal. Zeroed at WINDOW ASSEMBLY, not in BufferTempData: that output is cached PER BAR and a bar sits at slot 15 of one window and slot 0 of the next, so a slot-dependent value there would poison the cache or force a recompute per slot. The cache keeps true values; only this window's copies are cleared. Width contract untouched - same count, same positions - so conv/LSTM/HYBRID keep their bar-major rectangle unchanged and the block arrives at the newest bar, which for the LSTM is the final timestep. Zero-variance coordinates are safe through batch norm (divisor is MathMax(MathSqrt(var + BN_EPSILON), BN_MIN_STD)). Fingerprint gains |ALTW:1 when alt data is on. Same width and same .cfg, so nothing else would have caught a model trained under the replicated layout resuming under this one. Conditional append per the existing rule: configs without alt data keep their fingerprints and their trained models. NOT the concat branch. CNet is a strictly linear stack (CLayerDescription has no input-source field; NetBuild wires i to i+1 and stores layer L's weights on L-1), so a real two-tower model needs a new multi-input layer type across WarriorCPU, WarriorDML and the OpenCL kernels plus an .nnw format change - the highest-risk change in this repo, in the code that produced the transposed dense gradient, the Adam second-moment bug and the reversed LSTM window. This captures the part of that idea the measurement actually supports, at no engine risk. Compile-verified: 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
8ce635ce70 |
feat(altdata): external feature block wired into the NN feature window
- System\AltData.mqh: CAltDataPanel - publication-stamped CSV panel
(Common\Files\Warrior_EA\AltData\{SYM}_{TF}.csv), as-of lookup by bar
open, 0-fill degradation (mirrors cross-asset), hourly live refresh
- Topology: width block AFTER the .cfg name-list pin is pre-read
(ReadAltDataPinFromCfg) so a grown export can never mismatch a resumed
model's width or shift its slots
- Persistence: alt pin appended to the .cfg (append-and-length-guard
convention), adopt-don't-compare on load
- Features: emit block after Wyckoff SBI; EnsureFresh probe in
BuildFeatureWindow (never fires in tester)
- export.py: fixed a-priori scale constants (never data-fitted)
Widths change SP500 +4 / USDJPY +3 / XAUUSD +1 (fingerprint re-keys ->
fresh models on redeploy); EURUSD exports nothing and resumes unchanged.
Compiles 0 errors / 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
0adaea48b6 |
fix(resume): model reload stalled training - three hardenings on the resume path
A resumed META model hot-looped pass 1 (0->100% scan oscillation, silent for
3 minutes until the stall reporter fired) because EVERY window failed at the
first AD/Wyckoff feature: the init-time param adoption called
ReInitADIndicators unconditionally, destroying five freshly-calculating
indicator instances to recreate them with BYTE-IDENTICAL params (verified by
parsing the .nnw header - the MI tuner had kept the configured settings), at
process start, on a box with 1 GB free of 31. The replacements sat cold for
6+ minutes while full-history resweeps starved the indicator threads harder.
- AdoptIndicatorParams: installs a loaded param set into the tuner and
rebuilds handles ONLY when the set actually differs from what the live
indicators run. Both call sites (resume init + panel reload) use it.
- Resumed models get the same 3 warm-up passes as fresh ones. The skip was
the shared root cause of the cold-ATR (
|
||
|
AnimateDread
|
444909d0a3 |
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus
The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
923addf574 |
feat: pin the cross-asset pair set train->serve + warm the sync at init
The reference-pair set was re-discovered from Market Watch on every build, so adding or removing a terminal symbol silently changed what a trained model's six cross-asset features meant - the last open train/serve parity gap from the 2026-08-11 audit. The set a model's FIRST successful build actually used is now stamped into its .cfg (append-and-length-guard, adopt-don't-compare - the derived-barrier pattern) and every later build constructs the panel from exactly that list; a pinned pair that is temporarily unavailable is skipped, never substituted. Also warms SymbolSelect/SeriesInfo for every reference symbol at InitNeuralNetwork, so the terminal's ~minute of async cross-symbol download starts at init instead of when the first Build() trips over an unselected symbol - the source of the startup 'only 0 usable reference pairs' console failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
bd1037975a |
fix: the trailing incumbent read the future across eras; cold AD blocks cached zeros as truth
Three findings from the 2026-08-11 audit:
1. The excursion head's trailing-quantile ring was deliberately never cleared
between eras ("a rolling estimate of the market, not of the era") - but
pass 3 re-walks the SAME OOS window every era, so at each walk's restart
the ring still held the outcome masks of the newest OOS bars from the
previous walk: the chronological FUTURE of the bars about to be scored.
For the first ~window+horizon pushes of every era the "trailing" incumbent
was partly a leading one - conservative for the gate (an informed incumbent
is a harder hurdle) but exactly the self-made-artifact class
|
||
|
AnimateDread
|
f6150ee35b |
fix: cache only feature SUCCESSES - the cold-indicator poison came back through the guards ba13eef did not cover
|
||
|
AnimateDread
|
950b0fdab0 |
diag: name the cause when every feature window fails, and enforce the width contract
Era 0 stalls with "NOT ONE of 54681 scanned bars produced a usable feature window, windows ok=0 failed=54681" and nothing else. That line reads identically for a cold ATR, a conditionally-missing optional feature block and an out-of-range index, so it cannot be diagnosed without one restart per hypothesis. Two changes: 1. WIDTH CONTRACT in BufferTempData. Every enabled block must emit exactly m_neuronsCount values on EVERY bar. A block that emits its values on some bars and skips them on others (indicator, panel or series unavailable for that bar) does not merely shorten the window - it SHIFTS every feature after it into the wrong slot, and the net then trains on silently misaligned inputs that still look like a valid window to everything downstream. Now rejected, rolled back and reported once, naming the optional blocks (XA / SPR / swing context) as the ones carrying an availability test. Worth having independently of the current stall. 2. BuildFeatureWindow records WHICH lookback slot rejected and how much of the window was assembled, and the pass-1 stall report renders it: "slot 0 of 20 REJECTED (window had 0 of 760)" is an indicator warm-up or history-edge read; "every lookback bar ACCEPTED and the window was still short: 640 of 760" is a missing 6-value block. No behaviour change on a healthy run: the width check is an equality that already holds, and the diagnostics render only inside the total-failure branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
ba13eefecc |
fix: a resumed model cached a cold ATR as permanent, so it never trained
BufferTempData cached EVERY failure - m_featureCacheHasValue[idx]=true with m_featureCacheValid[idx]=false - and the cache never re-tries a miss. So a single feature read taken before the terminal had finished calculating the indicator buffers marked those bars unusable for the rest of the process, even though the data arrived milliseconds later. MT5 fills an indicator's buffers asynchronously after the handle is created, and a cold ATR returns 0 for EVERY index, not just its warm-up tail. BufferTempDataCompute rejects a bar with no ATR (correctly - the price features would be meaningless), so the whole window failed, and the whole cache was poisoned. Only resumed models were hit, because only they read features that early. Topology.mqh sets m_warmupPassesRemaining = netLoaded ? 0 : 3: a fresh start sits through three separately-scheduled Train() calls before anything touches a feature, which is exactly what those passes are for. A resumed one skips them and TuneIndicatorsAndTrain drives StartLabelCachePrebuild and the MI report from the first chart event. Its rationale - "a restart already has a proven-synced history" - holds for HISTORY and not for INDICATORS, which are recreated every process start. Downstream: BuildFeatureWindow failed on every bar of every era, so add_loop never went true, so pass 2, pass 3, the era counter and the checkpoint were all skipped and pass 1 swept 0->100% forever. The "0 samples" MI report line at startup was the same failure, four seconds earlier, already visible in the log. - a miss is now cached only when it is PERMANENT; the two "not ready yet" guards mark m_featureFailTransient and are recomputed on the next visit. Steady-state cost is ~ind_Periods bars per era, not 54k. - an era that discards itself now drops the feature cache before restarting, so any remaining cause of this state self-heals instead of looping. Deleting the .nnw "fixed" this only by turning the model back into a fresh one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
0c01dc279b |
feat: mini-batch gradient accumulation (F4), front-end-aware capacity budget (F6), split Wyckoff categoricals (N1)
Completes the 2026-08-09 training audit. FORCES A RETRAIN of every
Wyckoff-enabled config (N1 re-keys the fingerprint), and BOTH DLLs must be
redeployed alongside the .ex5 - they carry new exports.
F4 - mini-batch accumulation, TRAIN_BATCH_SIZE=32. Training was pure online
SGD (one weight update per bar), which is the mechanical source of the
era-to-era whipsaw every downstream guard was built to cope with. The O(n^2)
outer product is native - AccumulateWeightGrad / AccumulateWeightGradConv /
AccumulateBufferInto in Network.cl, WarriorCPU and WarriorDML - while the
optimizer step is host-side MQL5 shared by all tiers (ApplyAccumToBlock), so
there is one Adam/SGD implementation instead of four that can drift.
- the LSTM needs no outer-product kernel (WeightsGradient already holds the
sample's full dW) but could NOT simply be left un-zeroed between samples:
CPU_LSTMSeqBackward/DML_LSTMSeqBackward memset it on entry. Hence a
separate accumulator plus an elementwise add.
- batch-norm gamma/beta accumulate in host arrays, not new BatchOptions
slots - BN_OPT_STRIDE is baked into every persisted .nnw.
- scoped to pass 2; online learning keeps immediate updates. Every save /
checkpoint / scoring boundary flushes, scaling by the real sample count.
- degrades to per-sample updates (one log line) on a tier that cannot
accumulate, so old devices and DLL-free builds are unaffected.
- verified offline: DirectML/batch_accum_check.cpp drives the real exports
against an independent reference; at B=1 the accumulator matches the
shipped unbatched kernel's own gradient to 1.1e-16. Math only - the
in-situ check remains the per-layer dW/W report on a real era.
F6 - ComputeFirstLayerWidth budgeted against the RAW input width even where a
conv/LSTM front end had already reduced it, so an LSTM's dense stack was
charged for 1,280 inputs when it receives 64. Confirmed from the deployed
.cfg files: CONV, LSTM and HYBRID were all pinned at the 16-unit floor. Now
budgeted against the front-end output and capped at it (never fan out), with
the derivation reordered so both stages settle first.
N1 - EventCode/EventPhase/StructuralPhase are signed categoricals packing
direction and Wyckoff stage into one scalar across a sign discontinuity. Split
into direction + [0,1] magnitude, the same convention the base OHLC block uses.
Information-preserving; 13 readings now occupy 16 inputs.
Compiled clean (0 errors, 0 warnings); both DLLs rebuilt.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
33f106d99d |
fix: the indicator re-init leaked a terminal handle per candidate
This is what killed CONV and LSTM on 2026-08-07. Terminal journal:
19:19:40 6664 x "VirtualAlloc failed in large allocator"
19:19:40.829 expert Warrior_EA (SP500,H1) removed <- CONV
19:29:55 2048 x "VirtualAlloc failed in large allocator"
19:29:55.359 expert Warrior_EA (SP500,H1) removed <- LSTM
50ms and 71ms after each printed its "logit adjustment" line, i.e. the
instant era 0 tried to allocate its training queues. They did not hang -
MT5 shot them for running out of memory.
ReInitADIndicators() re-Create()s every enabled indicator and released
nothing. The comment above it asserted "CiCustom.Create() already
releases its old handle"; MQL5's CIndicator::Create is
m_handle = IndicatorCreate(symbol, period, type, num_params, params);
a plain overwrite, whose only success-path IndicatorRelease is in
~CIndicator. IndicatorRelease appeared nowhere in this codebase.
That function is the indicator tuner's inner loop. AutoTuneIndicators
scored 324 candidates per model on SP500 H1, so ~324 x 6 orphaned
terminal-side instances, each holding a full-history buffer set -
ADWyckoffEventStream is 14 buffers x ~38k bars x 8 bytes = ~4.3 MB each.
Gigabytes. Confirmed in the shutdown teardown, where a single surviving
expert still held 70 x ADWES, 53 x ADWyckoffEventStream, 23 x
WFS(48,3,1.80,1.10), 19 x ADMovingAverage, 18 x WYSB(162). Those
parameter sets are the tuner's candidate grids.
Release is UNCONDITIONAL, not gated on the handle having changed: MT5
refcounts instances by (symbol, period, params), so re-creating with
IDENTICAL params returns the SAME handle with the count incremented -
the "23 x WFS(48,3,1.80,1.10)" pattern. Either way Create() added one
reference and we hold one handle, so one release is owed.
Released AFTER the re-creates, never before: dropping the terminal's
last reference first would tear the instance down, so an identical-params
Create() would rebuild it from scratch instead of reusing the live one -
turning a refcount bump into a full recalculation over all history, 324
times over.
PAI was unaffected because it has no AD/Wyckoff indicators enabled (35
candidates, built-ins only). HYBRID survived on luck: CONV and LSTM died
2 and 12 minutes before its own sweep finished, freeing the memory.
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
bfc1da9de1 |
fix: the sequence models were reading the window backwards
BuildFeatureWindow() replaces eight hand-rolled copies of the same loop
and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first,
because MQL5 timeseries indices run backwards and `r + b` with b ascending
walks into the past.
Harmless for PAI and CONV - a dense layer learns a weight per position
either way, a conv learns time-mirrored kernels. Not harmless for the
recurrent stacks:
- LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t.
- It writes output[] only when t == steps-1: the visible output IS the
last hidden state.
- c_t = f*c_{t-1} + i*g decays toward the start of the sequence.
lstm_seq_flowcheck.cpp measured block 0's influence on the output at
1.2e-2 of block T-1's, at the shipped forget bias of 1.0.
So the bar being PREDICTED sat at the far end of the decay and the output
was handed to the OLDEST bar in the window - the exact inverse of what the
window is for. ~80x backwards on LSTM and HYBRID, on all three tiers
(OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never
surfaced as a backend discrepancy.
This does not create edge - the MI diagnostics read at the noise floor
(p=0.4975) with a working positive control. It makes the one hypothesis
those diagnostics explicitly do NOT cover testable: they are marginal and
per-bar, and state they "cannot rule out one that only exists in
combination or across time". The sequence model is the instrument for
across-time structure and it has been crippled, so that hypothesis has
never been honestly tested.
Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and
its features, so a stale .nnw would load cleanly and run a model fitted to
one ordering against the other, silently. Re-keying every config is the
point, not collateral damage. FORCES A FULL RETRAIN.
Also: the now-relative bar caches are re-keyed on the two live paths.
EnsureBarCachesCapacity() was only ever called from training paths, but
once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar
to RefreshConvergedSignal() and Train() is never re-entered - so nothing
cleared the feature cache again for the life of the process. A chart that
trained to convergence kept replaying the rows computed for the last
training era's bar grid: the live signal froze at its convergence-time
value, and OnlineLearnStep() backpropped those stale features against
freshly resolved labels. Backtests were never affected (an inference-only
process never allocates the arrays, so every read recomputes).
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
9e1c72aacc |
fix: make the indicator tuner actually measure, and gate what it installs
ROOT CAUSE of the zero spread measured on SP500 H1 2026-08-07 (all 17 candidates
returned exactly 0.00359 nats): the tune loop re-inits the indicators and then
scores, with no RefreshData() between.
ReInitADIndicators() does its part - Create() builds a NEW handle carrying the
new parameters, and the feature cache is flagged stale so features really are
recomputed. But BufferTempDataCompute() reads the CIndicatorBuffer objects, and
only Refresh() copies data out of a handle into those. So every candidate was
scored on values still held from the PREVIOUS handle. My earlier guess in the
diagnostic ("suspect the feature cache") was wrong: the cache invalidation works.
Two things land together, because neither is safe alone:
1. RefreshData() after the re-init, so a candidate is scored on its own features.
2. A SELECTION GATE on the install. bestScore is a MAXIMUM over candidates, and
the maximum of N draws from a null beats its incumbent almost every time - so
"it beat the incumbent" installs noise. This selector is the highest-stakes of
the three found in this audit because it ACTS: it overwrites the user's
configured indicator settings and forces BuildFreshTopology(), so the network
then trains on whatever the noise picked. Fixing (1) without (2) would have
made a dormant bug actively harmful.
The gate draws the winner's own permutation null once, then corrects the p-value
for having chosen it out of N with Sidak: p_family = 1 - (1-p)^N. Sidak rather
than the max-of-N resample used by the geometry scan because each candidate here
has a DIFFERENT feature set, so their draws cannot be pooled; Sidak needs only
the one null. Exact under independence, mildly anti-conservative under positive
dependence - stated in the comment rather than hidden. A rejected winner restores
the configured settings, which best[] cannot do since the descent mutates it.
Also reports the least-ready tunable handle's BarsCalculated(). IndicatorCreate()
calculates asynchronously, so if the spread is STILL zero the handles simply are
not done and the tuner needs to yield between candidates rather than score them
back to back - a state machine like the label prebuild. That distinction is now
readable from the log instead of requiring another guess.
No input, topology or label change: no retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
8c5ea639ee | feat: extend ADWyckoffEventStream with new range-lifecycle parameters and update related features | ||
|
AnimateDread
|
5f647ba5db |
fix: improve error messages and suppress false sharing-violation logs
- BufferDouble: replace hardcoded "DirectML/CPU-DLL" with dynamic backend name and add buffer index/element count to all error prints for easier debugging. - NetPersistence: distinguish missing file from transient lock by probing FileIsExist before logging, eliminating false "sharing violation" warnings when no saved model exists on first run. |
||
|
AnimateDread
|
ceb6342dfd |
feat(ai): spread as a volatility-regime feature, and fix a stale-index cache in both new blocks
Adds spread/ATR and the spread change ratio as network inputs (EnableSpreadFeature,
default on). Spread is the one microstructure channel that is both FX-available and
genuinely historical in the Strategy Tester - "during testing, the spread is not modeled
but is taken from historical data" - so unlike swap, signed tick flow or depth of market it
is something a backtest can honestly validate.
What it encodes, stated precisely because the raw measurement overstates it.
research/test_spread.py found spr/atr the strongest single feature in this codebase, on 5
of 8 instrument/geometry cells at 2-4x any volume feature. But the barrier LABEL charges
the spread inside its own barriers, so a wide-spread bar is mechanically likelier to
resolve as a loss and the feature would partly be predicting its own cost model. Relabelling
at zero cost and re-measuring the identical feature showed 20-40% of it WAS that tautology
and the majority was not (XAUUSD retained 97%). What survives is a volatility-regime
reading: spread is near-fixed while ATR is not, so the ratio runs high exactly when
realised volatility is below its own ATR estimate, which genuinely predicts whether
ATR-scaled barriers get reached. It is UNSIGNED - Neutral-vs-directional only, never a side.
Also fixes a stale-index bug I introduced with the cross-asset panel and had just repeated
in the spread series. Both cached on length alone:
if(m_crossAsset.Bars() >= bars) return true;
MQL5 series indices are relative to NOW, so one new closed candle shifts every index by
one. Keyed only on length, the panel keeps serving its index 0 as a bar that is no longer
the newest, and every cross-asset value is read one bar out of step with the price features
sitting beside it in the same vector - silently, with no error and no shape change. This is
the same class of defect as the dtStudied watermark behind the zero-direction backtests.
Both now carry a datetime anchor on m_Time.GetData(0), the same invalidation key the
label/feature bar caches already use.
And a performance fix that fell out of it: with correct invalidation the panel rebuilds on
every new bar, and RefreshConvergedSignal runs per bar - which in the tester would mean one
full multi-symbol resample per simulated bar at training depth. Inference only reads bars
0..m_historyBars-1 plus the panel's own slow window, so it now requests exactly that. The
cache check is >=, so a deeper panel left from training still satisfies it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
d80d9444a5 |
feat(ai): widen the volume feature block from 1 value to 4
The block fed exactly one number: (v[i] - v[i-1]) / v[i-1]. That is the first difference, and it cannot express three things that matter - the LEVEL relative to a baseline (two dead bars and two frantic bars both read ~0 change), and the two volume-vs-range interactions, where heavy participation that went NOWHERE (absorption) and heavy participation that travelled (continuation) mean opposite things and currently collapse onto the same value. research/test_volume.py measures each candidate's mutual information with the triple- barrier label across 3 instruments x 2 geometries, against a BLOCK-permutation null - blocks sized to the barrier horizon, because adjacent labels share almost their entire outcome window and a free shuffle yields a null so tight that everything looks significant. Finite-sample MI bias (~7/n here) is reported alongside rather than subtracted, since the permutation null already absorbs it. Result: volLevel beats the shipped change ratio outright on 4 of 6 cells (EURUSD 2:3 +0.000118 excess at p=0.006, USDJPY 1:2 +0.000284 at p=0.002); absorption is the single strongest reading anywhere in the sweep at EURUSD 1:2 (+0.000404, p=0.002) though it is null on XAUUSD; vol x range clears on 4 of 6. The shipped change ratio is itself significant on 5 of 6, so it stays. Kept OUT: a session-relative z-score against the same hour-of-day's own recent history. It was the weakest candidate - null on both EURUSD cells - and it is the only one needing per-hour rolling bookkeeping in MQL5. Not worth the state for a reading that did not survive its own null on the primary instrument. Magnitudes, stated plainly because they are the point: the excess MI is ~2e-4 nats against a label entropy near 1.05. That is under a tenth of one percent of the label's uncertainty. It is real, it repeats across instruments, and it is nowhere near an edge - this is worth having because it costs one 50-bar loop, not because it changes the answer. Prior work stands: the whole single-series feature family measured at the noise floor. m_neuronsCount is already in the fingerprint, so the width change re-keys existing caches by itself, which is correct - the input vector genuinely changed shape. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
8710240cd5 |
fix(signals): revive a dead MA model, and demote Sanyaku from state to event
Two defects surfaced by research/test_classic.py, both verified fixed by re-running the
transcription against 178k bars of EURUSD H1.
CSignalMA model 1 could never fire. For any recursive average - and MA_TYPE_EMA is the
shipped default - MA(i) = a*Close(i) + (1-a)*MA(i+1), so
DiffMA(i) = a * (Close(i) - MA(i+1))
DiffCloseMA(i) = (1-a) * (Close(i) - MA(i+1))
are positive multiples of one quantity and always share a sign. Model 1 asks for a close
BELOW a RISING average, which is precisely the combination that identity forbids: 0.000%
of bars, either direction, any symbol. The MQL5 standard library this was ported from
defaults to MODE_SMA, where the two are merely correlated - the bug arrived with the EMA
default, not with the port. Reading the slope one bar back (DiffMAPrev) breaks the tie for
every MA type while keeping the model's stated meaning. Now fires on 7.92% of bars.
CSignalIchimoku model 11 fired on 27% of bars at weight 100. Sanyaku is three standing
STATES conjoined with no transition term, so it held across long stretches - and being
last in the if-chain at the top weight, the module's highest-conviction reading was also
its most common one, overwriting all eight event models below it on a quarter of all bars.
The old comment rejected an event form because "demanding all three flip on the same bar
would fire almost never" - true, but that is not the alternative. Kouten is the TURN: the
ALIGNMENT transitions, and only one role need change for it to. Testing !Sanyaku(idx+1)
fires once per aligned stretch. Now 2.17%, in line with Kumo breakout (2.4%) and the
strong TK cross (1.1%). DataReady() extended one bar deeper to cover the lookback.
Neither pattern showed edge before or after; this is about the models meaning what they
say and the vote not being dominated by a constant.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
b4a704d309 |
feat(ai): triple-barrier labels replace exact-pivot ZigZag targets
The 31:1 class imbalance was self-inflicted by the TARGET, not a property of the market. Labelling only the exact bar where a ZigZag pivot confirms gave Buy 1164 / Sell 1164 / Neutral 35841, and every correction mechanism this codebase accumulated sits downstream of that one choice: the logit-adjusted loss and its range cap, the prior EMA, the +-3.0 output-bias seed, balanced-accuracy-then-precision selection with its coverage floor, the recall floor and its catch-22, the alternation gate, NMS, and the four oversampling designs that collapsed before them. The reference this engine is built on (references/neuronetworksbook.pdf ch. 3.1/3.3) also uses ZigZag, but targets the DIRECTION TO THE NEXT EXTREMUM on every bar - ~50/50 by construction, with no imbalance to correct at all. It never had this problem because it never asked "is this the pivot bar". Labels are now the triple barrier (Lopez de Prado ch. 3), using the EA's OWN SL_Mode/TP_Mode: does a trade opened at this bar's close reach its target before its stop, within a horizon. Buy = long resolves, Sell = short resolves, Neutral = neither. Consequences: - dir-precision in the era line stops being a proxy and becomes the win rate of the strategy under its own exit rules. - Expected balance ~25/25/50 at the shipped 1:3 (gambler's ruin), i.e. ~2:1 instead of 31:1. Measured and logged at the end of the prebuild. - Spread is charged on both legs, so it is a NET win rate. - Intrabar ambiguity resolves to the STOP. OHLC cannot order two touches inside one bar and the optimistic reading is how a backtested edge becomes a live loss. ZigZag stays as input features (EnableSwingContext) and now also supplies the vertical barrier: the horizon is the median confirmed leg length, snapped to a coarse ladder. Derived, not configured, and deliberately kept out of the filename fingerprint - a filename keyed on a measured quantity orphans a trained model the moment the measurement moves. Removed, because the premise died with the old target: - the alternation gate. Correct for pivot labels (a ZigZag cannot emit two same-type pivots in a row, so a repeat was provably a false fire), and wrong for barrier labels, which answer each bar independently. It also took its worst consequence with it: a one-sided model previously got ONE trade per backtest, a hard blocker on marketplace validation. - SignalClusterWindow now defaults off - it de-duplicated repeats that are now real trades. Kept as an opt-in display control. - LABEL_WINDOW_BARS, the pivot-widening pass, ConfirmedZigZagLabel. - the era-0 output-bias seed now needs a genuinely dominant class (0.70) rather than 0.40; at ~50% Neutral a +-3.0 seed is a distortion, not a correction. Also fixed, both found while wiring the above: 1. RefreshConvergedSignal sized its buffers from a date delta (Bars(sym, period, dtStudied, TimeCurrent())). dtStudied is a training watermark; in the tester it is loaded from a live-chart save AHEAD of the simulated date, so the interval inverted, Bars() returned ~0, and the buffer came out at exactly m_historyBars - deep enough for the OHLC window and far too shallow for the Donchian-50 / 20-bar-return / SMA extension behind it. Inference silently computed DIFFERENT features from the ones training learned on, live as well as in the tester. Now sized from what the feature builder actually needs. 2. The barrier horizon is resolved on the deployed path too. A deployed model never enters Train(), so it never reached the prebuild, and OnlineLearnStep reads the horizon as its confirmation delay - left at the fallback it would have backpropped bars whose barriers had not resolved. Silent lookahead in the one place that writes to a live model. SL_Mode/TP_Mode join the weights fingerprint: they define the labels now, so a model trained at 1:3 must never be silently reused at 1:1. This re-keys every pre-existing model by design - none were trained on this task. Inference census extended with the vote gate. LongCondition/ShortCondition open with a readiness check the refresh counters never see; in the tester it reduces to "the seeded _optcache.nnw must have LOADED", and if it did not, every vote is hard-zeroed while the model still answers Buy. The old three counters would have read that as "the model says Neutral" - false, and a completely different fix. This is the leading candidate for the zero-direction backtest and the census can now name it in one run. Both builds compile 0 errors / 0 warnings. Forces a full retrain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
26ac479bae |
docs: refactor + audit notes; fix malformed comment banner
REFACTOR_NOTES.md records what was found, what was changed, what was deliberately left alone, and the one investigation that is still open (the MLP CPU-DLL slowdown, with the parameter counts that rule out my earlier "largest weight matrix" explanation). Also restores the missing opening rule on ReInitADIndicators' comment banner. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
2de93539d4 |
refactor: split CExpertSignalAIBase implementation by responsibility
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87 method bodies covering training, labelling, feature extraction, persistence, chart drawing, online learning, the GA auto-tuner and inference, all in one file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling past the era loop. Moved the bodies into Expert\AIBase\, included at the bottom of the original after the class declaration: Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy Features.mqh 1093 indicator creation + per-bar input feature vector ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild AutoTune.mqh 275 genetic tuner (population, crossover, halving) Inference.mqh 235 softmax, prior calibration, class priors ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only) This is a pure relocation - verified mechanically, not by eye: HEAD's file reconstructed from the eight partials plus the surviving remainder is byte-identical to HEAD, span for span (scratchpad verify_split.py). No declaration moved, no signature changed, no code rewritten, so behaviour is unchanged by construction. Compiles 0 errors, 0 warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |