forked from animatedread/Warrior_EA
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
AnimateDread
|
1bf3eba68a |
feat(meta): self-contained corpus - the META chart sweeps the real classic ladders over its own history
The user should not need a tester corpus run per symbol. Every pattern
condition in Signals\Signal{MA,RSI,MACD,Ichimoku}.mqh anchors its reads on
`int idx = StartIndex()` with zero hardcoded indices (verified), so a
name-hiding StartIndex override + EvalShift(i) on CExpertSignalCustom makes
the EXACT live ladder code answer "what would you have fired at bar i" -
the silent-divergence trap that justified the DB corpus does not exist on
this path, and neither do the GMT-offset ambiguity, the DB row caps, or
the wipe procedure.
- CExpertSignalCustom: m_evalShift + StartIndex()/EvalShift() +
SweepPrepare(bars) (deep-resizes the shared price series); the four
classic signal classes override SweepPrepare to deep-resize their own
indicator buffers.
- CSignalMETA::BuildCorpusBySweep: per bar x per source filter, run
Direction() shifted, harvest the per-side pattern slots + netVote into
the same corpus arrays the DB loader fills; entry=bar open so
MetaPrepareEra's resolution matches at offset +0 with zero price error.
DB corpus remains the fallback when classic filters are disabled.
- Warrior_EA.mq5: META gets the enabled classic filters as candidate
sources (family ids match the descriptor one-hot).
- UseDatabaseRanking default false -> true (user request): a META chart
journals + ranks out of the box.
Workflow per symbol is now: attach ONE chart with AIType=META (optionally
Meta_ExportDataset=true for the offline pool) - candidates, labels,
training and export all happen in place, ~10 seconds of sweep instead of a
tester run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
444909d0a3 |
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus
The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
68459208bf |
fix(meta): make the stale-DB corpus warning unmissable in the tester
The warning lived inside the VerboseMode-gated corpus report, so a forgotten wipe silently voided an entire 18-year corpus run - the outdated-row guard rejected the whole replay against leftover rows and the run appended 35 rows instead of building a corpus. The check now runs unconditionally at tester OnInit (MetaCorpusStaleCheck): 52 quiet one-row newest-key probes vs the test start, with a loud stop- wipe-rerun instruction when the DB is newer than the test. Absent tables probe quietly via FetchNewestTimeKey''s new quiet flag. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
4507ea69a9 |
feat(meta): S1 - the signal DB becomes the meta-label training corpus
Implements stage S1 of Meta_Labeling_Design.md, superseding the original "training-time ladder sweep": the per-side journaling from 652bf81/195be20 already produces the exact candidate stream a sweep would compute - every pattern instance the live ladders fire, both sides, uncensored, with netVote and touchable entry price - so the corpus is READ from the DB instead of re-implementing 26 ladder conditions in training code. That eliminates the silent-divergence trap outright: the corpus is by construction identical to live behaviour. Accepted costs are documented in the module and the doc: coverage equals the populating backtest, and sampling is one candidate per fire-stretch (the right dedup for training anyway). - Expert\AIBase\MetaCorpus.mqh: CMetaCorpus reader (52 tables -> SMetaCandidate rows) + VerboseMode OnInit report: volume/closed/ S&R-win-rate per family, span, and the GMT->server bar-offset match table (offsets +0..+3h) that S2''s label plumbing pins to - measured, not assumed. - DB_MaxRowsPerTable input (default 1000 = old MAX_TABLE_ROWS): a corpus build raises it (e.g. 20000) so a 15-20 year backtest isn''t pruned; wired through CExpertSignalCustom::MaxTableRows(). - Report-only stage: nothing downstream consumes the corpus yet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
652bf81112 |
fix(db): per-side pattern journaling + versioned journaling semantics
The labelMatchesVote gate compared a single last-writer-wins label (LongCondition then ShortCondition) against the net vote sign, which structurally censored the pattern tables: a long event co-occurring with any short-side state model lost its label to the later writer and was dropped, while the mirrored short event journaled fine. Ichimoku models 0/3 and MA model 1 could not produce a row at all by construction (MA model 1 was "revived" in |
||
|
AnimateDread
|
36e8463310 | refactor: derive history bars for input sequences and update related configurations | ||
|
AnimateDread
|
77e8080cfe |
fix: four risk-layer holes a funded account would eventually find
1. The expectancy stop was stone dead at shipped defaults. Its only feed -
RecordTradeResult inside CTradeJournalManager::Update() - ran solely under
UseDatabaseRanking, which ships false, so the
|
||
|
AnimateDread
|
c5acc5a7a8 |
perf: pass 1 forward-passed ~40% of bars that a later pass redid anyway
Pass 1 already skipped its feedForward on QUEUED bars, because pass 2 redoes them. The same argument covers two more bands it was still forwarding: OOS window (30% of bars) - pass 3 re-forwards every one of them calibration band (~10% of bars) - pass 2.5 re-forwards every one of them All three passes derive their bounds from the same helpers and apply the identical eligibility test, so the bar sets are equal by construction, not by coincidence. Only the two purge bands and the ineligible edge bars are visited in pass 1 and nowhere else - those keep their forward pass. The scan's copy was never the one that survived. Its arrow-cache write was overwritten by pass 3's (with the thresholded, post-training decision), its status-label paint was transient, and its predicted-class tally measured last era's weights. Those tallies move to pass 2.5 and pass 3, on the raw argmax exactly as pass 1 and pass 2 count it, so the population behind the panel's "Predicted -> Buy/Sell/Neutral" line is unchanged and stays comparable with the "Actual" line beside it, which pass 1 still accumulates over every labelled bar. Verified unaffected by the cut: dPrevSignal and m_lastBarTime are both written last by bars 0/1, which are label-ineligible and therefore still forwarded, so FinalizeTrainRun's `dtStudied = m_lastBarTime` and Lifecycle's newBarPending sentinel read the same values as before. Correctness, not just speed: batch norm is UNFROZEN during pass 1 (passes 2.5 and 3 freeze it deliberately), so every scan-time forward on a held-out bar was advancing the BN running mean/variance from data the model is graded on. Those running statistics are inference-time model state. It is the mild, unsupervised kind of leakage - feature statistics, not labels - but it fed the weights pass 3 then scored, and it is now gone. Cost: ~40% of all bars lose one forward pass per era, ~16% of net time once pass 2's backward pass is weighted in. Per-dispatch, so it lands on every backend. Both variants compile 0 errors / 0 warnings. Build tag scan-nofwd-v5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
e2c959331f |
perf: the excursion head cost 3.6x era time - cut its dispatches ~250x
Measured on exc-race-v3: LSTM era 300s -> 1087s (net 272->748s, "other" 30->337s). My estimate had been "single-digit percent". The cost is per-DISPATCH, not per-FLOP, and therefore hits EVERY backend: the head is 19k weights and ~2.4 GFLOP an era - seconds of arithmetic - but ~48k forward/backward calls x several layer submits each, and its 760-wide layer exceeds the CPU DLL's inline threshold so each one pays a real handoff. The classifier's own net time tripled too, from contention with a second pool on an already-full box. Three changes, all backend-neutral because they remove submits rather than tune threads: SCORE ONLY DISJOINT WINDOWS (~64x). Adjacent bars share all but one bar of their horizon, so 16k consecutive bars were always ~250 independent observations - the full-sample tally was never worth more than the disjoint one, it just quoted an n that was ~64x too large. Dropping it costs nothing statistically and removes 63 of every 64 forward passes. The two parallel tallies collapse into one, which is also less code. The trailing ring still advances on every bar: it needs the outcome SEQUENCE, and that is array lookups, not a forward pass. TRAIN ON EVERY 4th PRIMARY BAR (4x). The target is low-dimensional and strongly autocorrelated - neighbouring bars carry near-identical excursion information - so per-bar training buys resolution the target does not have. Strided on ATTEMPTS, not acceptances, so a stretch of unlabelled bars cannot silently change the spacing. OWN TIMING COLUMN. The head's passes were landing in the era line's "other" bucket, which is how a 3.6x regression read as an unexplained jump in the one column nobody attributes. A cost that cannot be seen in the timing line cannot be traded off against anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
345a672500 |
fix: purge every EA object namespace on init and after deinit teardown
Leftover objects survived deinit because the cleanup list had drifted. PurgeChart()'s own comment said it removed "our namespaced signal arrows plus the status-label objects" while the code removed arrows ONLY, and the panel prefix was swept at OnInit and nowhere else - so an ordinary deinit left the status line, and any panel straggler, on the chart. Three scattered call sites and a comment cannot be kept in step. There is now ONE list - WarriorChartPrefixes() - covering arrows, status label and panel, and one sweep, WarriorPurgeChartObjects(), used by every path. Add a prefix there when a new object family appears and every cleanup picks it up. Two call sites added: OnInit, before ANYTHING is drawn (including the status label it would otherwise delete). Chart objects live in the chart PROFILE, not in the EA, so they outlive the process: a deinit force-terminated at MetaTrader's ~4,500 ms budget, a crash, a terminal kill, or an .ex5 replaced while attached all strand objects no later deinit will ever own - and deleting the EA's files does not remove them, which is why they read as corruption. Arrows are included: LoadChartSignals restores them from their sidecar moments later and already opens with its own arrow sweep, so this only removes orphans the sidecar does not account for - the ones SaveChartSignals would otherwise ADOPT, since it rebuilds that sidecar by scanning the chart. OnDeinit, after ExtPanel.Destroy. Destroy walks an unbounded control tree and ClearStatusLabel clears text rather than guaranteeing object removal; either can leave a straggler and nothing looked afterwards. Bounded work - three prefix deletes and one object-list scan - so it respects the ordering rule that keeps the cheap visible cleanup ahead of the heavy save. Arrows excluded: ShutdownChartCleanup already persisted and removed them and re-deleting would race that write. The two are complementary: the deinit sweep closes the ordinary case, the OnInit purge closes the case where MetaTrader never let us finish. Only the second can help after a starved shutdown. Both sweeps rescan by name across EVERY object type and delete what the bulk call missed. ObjectsDeleteAll's return has already been observed disagreeing with a by-name scan of the same chart microseconds apart, and object commands are queued on the chart rather than applied inline, so a returned count is not evidence the objects are gone. Panel create site now uses WARRIOR_PANEL_PREFIX instead of a literal, so the name cannot drift away from the list that cleans it up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
4cfbb82634 |
feat: race the excursion head against a trailing-quantile incumbent
Beating a frozen global constant is the weakest admissible bar for replacing a global constant. The honest incumbent is a rolling rung frequency: it adapts to the volatility regime - exactly what the head claims to predict - and needs no model, no 760 inputs and no training. Implemented as a ring of per-bar outcome bitmasks (32 rungs fit one ulong), sized horizon + EXCURSION_TRAIL_WINDOW. The newest `horizon` entries are held back UNRESOLVED: a bar's rung outcomes are only known one horizon later, so using them would be lookahead and would flatter the incumbent into an opponent the head could never fairly beat. Pass 3 walks oldest-to-newest, so "pushed more than horizon bars ago" is exactly "resolved by now". Each push is O(rungs), not O(window). The head's decision-rung Brier is pro-rated to the trailing estimate's coverage before the ratio, since the incumbent only scores bars where its window is warm. This line is worth reading on its own, independently of the head: if the trailing quantile beats the global constant, that is a cheap risk-control win available with no machine learning at all - and it is the same number either way, so the run answers both questions in one pass. The ring is deliberately NOT reset per era - it estimates the market, not the era, and re-warming 500 bars every era would leave the incumbent unusable over the first chunk of every scoring pass, handing the head a free win on exactly those bars. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
06d4785e39 |
fix: the excursion gate would have passed Stage 2 on an artifact I made
Second-opinion review killed the +4.2% far-rung result, correctly, and
the mechanism is my own bug. A head trained toward {0.05,0.9} converges
to 0.05+0.85p, so its bias is 0.05-0.15p: negative where p is near 1,
POSITIVE where p < 1/3, growing monotonically as the rung gets farther.
Against a baseline frozen at the IS rate, an upward-biased head scores
positive Brier skill whenever the OOS rate merely sits above the IS rate.
Predicted signature: huge negatives near, ~zero at p=1/3, growing
positives far. Observed: -82% ... -0.6% ... +1.2/+2.7/+4.2. The far rungs
were not the clean end of a distorted measurement, they were the other
face of the same artifact. Everything before
|
||
|
AnimateDread
|
25aca8367c |
fix: the excursion head was scored against a cap I gave it
ExcursionTargets built its 32 binary targets from the classifier's LABEL_SMOOTH_HIGH/LOW (0.9/0.05). That caps what the head can ever output at 0.9, and the near ladder rungs have base rates close to 1.0 - almost every bar travels 0.5 ATR inside a 64-bar horizon. The Brier comparison is then decided before the net learns anything: constant at 0.99 -> 0.99*(0.01)^2 + 0.01*(0.99)^2 = 0.0099 head at 0.90 -> 0.99*(0.10)^2 + 0.01*(0.90)^2 = 0.0180 skill -82% Which is what the first run reported at rung 0.50: PAI -61.8%, CONV -146%. A property of the target encoding, not of predictability. Smoothing earns its place on the 3-class head, where it stops one logit running away inside a softmax competition. There is no competition here and this head is scored on calibration, so it has to be free to say 0.99 when the answer is 0.99. Hard 1/0 is safe against the runaway smoothing guards: this is an MSE-on-sigmoid gradient (calcOutputGradients) whose (target - output) term vanishes as the output approaches the target, not the unbounded-logit cross-entropy the classifier uses. The far rungs, where the artifact is smallest, already showed positive skill on the two topologies with a sequence stage (LSTM 3.00:+1.2% 4.00:+2.7% 5.00:+4.2%, HYBRID similar), so the verdict was being decided by the most distorted end of the ladder. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
f6150ee35b |
fix: cache only feature SUCCESSES - the cold-indicator poison came back through the guards ba13eef did not cover
|
||
|
AnimateDread
|
950b0fdab0 |
diag: name the cause when every feature window fails, and enforce the width contract
Era 0 stalls with "NOT ONE of 54681 scanned bars produced a usable feature window, windows ok=0 failed=54681" and nothing else. That line reads identically for a cold ATR, a conditionally-missing optional feature block and an out-of-range index, so it cannot be diagnosed without one restart per hypothesis. Two changes: 1. WIDTH CONTRACT in BufferTempData. Every enabled block must emit exactly m_neuronsCount values on EVERY bar. A block that emits its values on some bars and skips them on others (indicator, panel or series unavailable for that bar) does not merely shorten the window - it SHIFTS every feature after it into the wrong slot, and the net then trains on silently misaligned inputs that still look like a valid window to everything downstream. Now rejected, rolled back and reported once, naming the optional blocks (XA / SPR / swing context) as the ones carrying an availability test. Worth having independently of the current stall. 2. BuildFeatureWindow records WHICH lookback slot rejected and how much of the window was assembled, and the pass-1 stall report renders it: "slot 0 of 20 REJECTED (window had 0 of 760)" is an indicator warm-up or history-edge read; "every lookback bar ACCEPTED and the window was still short: 640 of 760" is a missing 6-value block. No behaviour change on a healthy run: the width check is an equality that already holds, and the diagnostics render only inside the total-failure branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
2d28f6542b |
feat: excursion-size head (Stage 1, measurement only)
Direction is closed - normalised asymmetry fails on three instruments with a working positive control, and the classifier's own best-of-999 era-cap test agrees (+0.9pp = 1.48 sigma, family-wise p=1.0000). SIZE is a different question and RANGE clears at ~4x its null. Checked the denomination before building on that, since the source memo warns to: m_excUpCache holds (maxHigh - fill)/ATR, so "RANGE is predictable" is a claim about travel RELATIVE to current ATR, not a restatement of "ATR is autocorrelated". It is exactly the part a fixed multiple (stop 3.31*ATR, target 1.64*ATR) discards. A second small CNet, 760 -> 24 -> 32 sigmoid outputs = P(price reaches ladder rung k) upward and downward. Survival parameterisation rather than regressing the multiple, because it needs nothing new from CNet: sigmoid outputs and the per-neuron delta the `total != 3` branch already applies (a quantile head would need a linear activation and a pinball gradient in Network.mqh, Network.cl and the DirectML path, on a class four topologies share). Targets are free - m_ladderUpAt already records first-touch age per rung with 0 meaning never reached. Separate net, not extra outputs on the classifier: more outputs would change m_outputNeuronsCount, the .nnw shape and the fingerprint, and push the count off 3 - the exact condition backProp uses to select the joint softmax gradient the 3-class head depends on. The classifier is bit-for-bit unaffected and this is removable without trace. STAGE 1 PLACES NO ORDERS. It reports a Brier skill score against the constant per-rung base rate - the baseline a fixed ATR multiple already assumes - with both predictors fitted IS and evaluated OOS, so neither gets a look at the test set. Positive skill justifies Stage 2 (drive SL/TP and sizing off ExcursionQuantile, which is defined and deliberately uncalled). Zero or negative means ATR already carries everything and Stage 2 must not be built. Trains only on primary occurrences: the replay queue oversamples for CLASS balance, and a direction-balanced sample is a biased SIZE sample. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
0c8b4dc30d |
fix: the deploy gate graded the un-thresholded model
coveragePct, dirPrecPct and the declustered TRADED tally were all computed from oPrevSignal - the RAW argmax - while the live order, the arrow and the panel all run on oDeploySignal, which is argmax AFTER the confidence threshold. The gate was certifying a strategy the EA does not trade. Invisible until now: the threshold sat at ~0.02, so the two populations were the same set. The held-out calibration slice ( |
||
|
AnimateDread
|
2189316c35 |
fix: the operating point was fitted on bars the net had memorized
FitDirConfThreshold harvested its margin histogram from pass 2's own
backprop samples. Pairing every fit against the same era's OOS result
shows what that measured:
PAI era 1 IS 25% cov @ 66.1% (-0.8pp) -> OOS 64% (-3pp) gap +2.1pp
PAI era 76 IS 90% cov @ 79.6% (+12.7pp) -> OOS 65% (-2pp) gap +14.6pp
LSTM era 9 IS 77% cov @ 81.6% (+14.6pp) -> OOS 63% (-4pp) gap +18.6pp
The gap grows monotonically while OOS stays flat, so within a handful of
eras the curve stops describing behaviour on unseen bars. That is fatal
here specifically, because the objective branches on the SIGN of
(p - break-even): the memorized curve reads +12pp at 95% coverage, so
coverage x (p - p0) correctly maximises coverage and returns ~0.02 - fire
on every bar. The "p < p0 -> get more selective" branch, which is the
actual regime and the entire point of
|
||
|
AnimateDread
|
d919a4aea2 |
feat: 10-bar decluster window + alternation on every signal consumer
SignalClusterWindow 3 -> 10 for all topologies. On H1 a 3-bar window
collapsed only the tightest runs and left visible clusters at every
turn; 10 bars is closer to the spacing of genuinely distinct setups.
ALTERNATION. Rule 1 only collapses a same-direction run INSIDE the
window; past it a second Buy is emitted with no Sell between, giving
Buy/Buy/Buy/Sell. With both directions tradeable that sequence is the
model re-entering a move it is already in rather than finding a new
one. The kept sequence must now alternate: the first signal passes,
and after that a direction passes only if the last KEPT signal was the
opposite one.
Added to ALL THREE consumers, with identical logic, because they must
agree:
- NmsLiveAccept -> the live trade
- pass 3's OOS replay -> the tally the deploy gate grades
- PruneDirectionalClusters -> the drawn history
A rule applied to only some of these certifies one strategy and trades
another - the same defect class as the geometry the gate certified
while OpenParams placed something else (
|
||
|
AnimateDread
|
ccfbc62561 |
fix: the recall gate was unsatisfiable and the LR decay was a spiral
Both made the run structurally unable to succeed, independently of any
signal in the data. Found by reading the 13:01 log.
RECALL GATE. m_objectiveMet required Buy, Sell AND Neutral OOS recall
each >= 40%. First-touch resolution (
|
||
|
AnimateDread
|
ece2154102 |
fix: flush the in-flight era on shutdown; sweep orphaned chart objects on attach
Chart objects live in the MT5 chart PROFILE, not in this EA's files. They survive a terminal restart, a recompile, and deleting every .nnw/.cfg/.stats/.arrows on disk. Only a deinit that RUNS TO COMPLETION removes them - and MetaTrader force-terminates OnDeinit at roughly 4,500 ms, so a run killed mid-cleanup orphans them permanently with no owner left to clean up after. That is the "deleted every file, recompiled, restarted, old arrows and a stale panel still there" report: nothing was wrong with the files and deleting them could not have helped. Both halves are fixed. STOP OVERRUNNING THE BUDGET. OnDeinit used to finalise the in-flight run (StopTraining -> FinalizeTrainRun: checkpoint restore, live-state re-seed) and then write two full nets per chart. On four charts that is the bulk of the budget, spent to preserve a PARTIAL era that was never scored, never checkpointed and never deployable. FlushTrainRun() discards it instead - drop the resumable bookkeeping, leave the net neutral (unfreeze BN, flush the batch, batch size 1), skip the save - and training resumes from the last completed era, which the era-end save and the periodic autosave have already put on disk. What is discarded is bounded by one era. A CONVERGED model keeps the old finalise-and-save path: its weights can carry online-learning updates made since the last era boundary, and for a deployed model no further era boundary is coming to persist them. MAKE CLEANUP SELF-HEALING. Every purge sat behind a branch - no model loaded, sidecar missing - so the common paths returned leaving whatever the previous instance stranded. LoadChartSignals now sweeps the arrow namespace unconditionally before restoring, so the post-init chart holds exactly what the sidecar holds whichever branch runs, and the panel gets the same treatment before Create() (CAppDialog namespaces its controls, so a killed Destroy strands the lot and the next attach draws a second panel on the corpse). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AnimateDread
|
1a5157befc |
fix: training could only advance one 120ms chunk per bar
ScheduleTrainingIfNeeded() armed the next Train() call only when dtStudied < lastBarDate. That watermark test is right for a CONVERGED model - one inference refresh per new bar - and wrong for a training run, because Train() is chunked: it does ~120ms of work and yields, needing thousands of calls to finish one era, and every one of those calls has to be armed from there. dtStudied is two incompatible things. Train() sets it to the training WINDOW START (~2008); FinalizeTrainRun() sets it to the last bar SCANNED (~now). So the moment any run finalized, the scheduler went silent until the next candle closed. On H1 that is one chunk per hour. The symptom was indistinguishable from a hang: no era lines, no heartbeats, not one of the six instrumented stall branches - because Train() was not being CALLED. The TRAIN STALL line that caught it reported runActive=Y only because m_trainRunActive had been set microseconds earlier in that same call, and eraResume=N proved no era was in flight. Two log bursts, 28 minutes apart, exactly one H1 bar. Before |
||
|
AnimateDread
|
9a7c37f334 |
fix: live trades now use the geometry the gate certifies; perf: BN kernels
Three changes, one theme: the trade placed, the trade graded, and the trade computed are now the same trade. 1) GEOMETRY WIRE (correctness, the ranked #1 open issue). The measured barrier pair reached the LABELS only - OpenParams still placed orders at the enum geometry (2*ATR/6*ATR), so the deploy gate certified "reaches 1.62*ATR before 3.33*ATR above break-even" about trades the EA never placed. Published via g_DerivedSlAtrMult/g_DerivedTpAtrMult (ConfidenceBridge, same same-tick contract as the confidence globals, because OpenParams runs on the root signal which has no pointer to the AI filter). Two writers: DeriveBarrierGeometry at era 0, and the .cfg adoption a deployed model takes. Overrides both legs and both Intelligent modes - the certificate is exact or it is nothing. TP is ATR-anchored like the label, NOT risk-relative, so a floor-widened stop cannot reshape the certified target. 2) BATCH NORM RUNS DEVICE-SIDE ON OPENCL. Four kernels in Network.cl - forward, hidden gradient, gamma/beta accumulate, gamma/beta apply - each a line-for-line transcription of the host implementation (NormalizeHost / HiddenGradHost / StepGammaBeta) including every NaN guard, clamp, and the exact moment-write ordering. The host copies remain the runtime for the DLL and pure-MQL5 tiers and the reference the kernels must match. Because this box has no OpenCL platform, the safety story is layered: - shim validation: kernels compiled as C and driven against a fp64 host transcription over NaN-poisoned stats, NaN gamma, over-clamp inputs, the frozen path, both optimizers, 3 batches - ALL PASS, worst normalized diff 0.132 vs tolerance 1.0 - in-situ self-check: each kernel is compared against its host twin ON FIRST USE on the real device (SelfCheckBn*), covering what the shim cannot - arg indices and buffer bindings. Any disagreement resyncs from the good copy, latches all BN kernels off process-wide, and training continues host-side. A transcription bug costs a warning and some speed, never a poisoned .nnw. - sync discipline: BatchOptions is now a CBufferDouble with explicit authority tracking (m_bnDeviceAuthoritative). Checkpoints/saves pull read-only; restores/loads/resets push; a mid-batch handover drains the device gamma/beta accumulator into the host arrays so no sample is lost. 3) SMALL FIXES. Apply-kernel build failure now latches the dispatch path at init (one warning instead of warning + failed Execute). Build tag bumped to win-scoring-gpu-v1 - first tag change since expectancy-stop-v1 despite five binary-changing commits. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
217b9bc9bf |
feat: remove Min_Risk_Reward_Ratio - a guess was overriding a measurement
The barrier geometry is derived from the instrument's own excursion distribution (stop at q75 of adverse travel, target at q50 of favourable), and then a 1:2 floor was applied on top, raising the target to twice whatever the stop happened to be. On SP500 H1 that pushed the target to 6.66*ATR, reached on 3.3% of bars inside the horizon - so the label became "almost never a win" and every topology was trained to predict an event that essentially does not occur. A measured target has to stay measured. The ratio never bought what it was believed to buy. A reward:risk floor does not create expectancy; it trades hit rate against payoff at a break-even the geometry already fixes - which this project has separately MEASURED (payoff 0.92 -> 5.72 with expectancy flat). What it did buy was two outages: four consecutive Market validation rejections for "no trading operations" when it rejected 100% of setups, and the label corruption above. Removed: - the input and the RISK_REWARD_RATIO enum (deleted, not left dangling - a live enum with no input behind it is the shape of the stale-.set incident that trained ~250 eras on the wrong target) - the forced target raise in the label geometry - the rrOK eligibility gate in the barrier-geometry scan, so every unclamped pairing now competes on the measurement alone. Clamping stays disqualifying for its own unrelated reason. - the reward < minRR*risk veto in OpenParams Kept: g_TradeRewardRiskRatio still computed and still bridged to Kelly sizing in MoneyIntelligent - the ratio as a SIZING input was always the sound use. Risk stays bounded where it actually is - account risk % and CRiskBudget. The low-reachability warning survives but is re-aimed: with nothing inflating the target, a target the market rarely reaches can only mean the horizon is truncating the excursions the geometry is derived from. Both build variants compile 0 errors / 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
371f8aaecd |
fix: the Adam second moment was never Adam - all four tiers
Root cause of the B=32 regression, and it predates F4 entirely. Every Adam
kernel stored v already square-rooted and then fed that stored value back in
as if it were the variance:
v_new = sqrt(b2 * v_old + (1 - b2) * g^2)
That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below
unit scale, so the denominator stops tracking the gradient and Adam degrades
into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll
(batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a
constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where
a scale-invariant optimizer gives the same distance for both. After the fix
all six magnitudes read 1.199 and v tracks |g| exactly.
It hit conv/LSTM specifically because they sit behind a batch-norm with
running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep
in the degraded regime - while the dense stack near the loss stayed in the
working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/
0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already
squared v back for gamma/beta and its comment named the kernels as wrong,
which is exactly why gamma/beta kept training while the stages behind froze.
Persisted .nnw needs no migration - v keeps its std-dev meaning.
Also, the two ways F4 exposed it, both mine:
- No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods
(Krizhevsky 2014; Granziol et al. 2022), applied once in
InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD.
- Plateau patience denominated in eras, so raising B made the ladder 32x more
impatient in its only unit. PAI converged at era 41 on ~49k updates where
the same config had been finding new bests at era 1028.
TrainPlateauPatienceEras() stretches it by the same sqrt(B).
TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23
eras per stage, not 8 -> 45). Both helpers are identities at B=1.
Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The
perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy
alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into
the ranking key rather than checked at deploy time so a one-sided era cannot
become best-so-far in the first place.
Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded
CAppDialog teardown that sat ahead of it - the same ordering inversion the
rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit
(vs ~1.1 s for the three that finished) having reached none of its cleanup,
so its arrows stayed on the chart. Steps are now timed in the log.
PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot
as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now
walks every object type and reports the object counts when both are zero.
Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt.
FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
da54639996 |
feat: expectancy stop - halt when the measured result says the strategy loses
The daily (4%) and total (8%) rules bound how FAST an account can lose. Nothing
noticed WHETHER it was losing. A negative-expectancy signal traded at 1% inside
that envelope breaches no rule and still arrives at zero - it just takes longer,
with every limit green the whole way down. That is the realistic way this EA
destroys an account, and no existing guard could see it.
THE ARITHMETIC THIS ENFORCES. Expected value per trade is p*TP - (1-p)*SL - cost.
With no directional edge p equals SL/(SL+TP), which is also the break-even rate,
so the payoff terms cancel exactly and EV = -cost. Expected P&L is -(trades) x
cost: strictly negative, proportional to activity. Measured here: directional
precision 23-24% against a 25% break-even, flat across every confidence tier,
with 58 points of spread on SP500. Sizing, stop placement and trailing move
variance around that mean; none of them changes its sign.
So every closed position now reports its result in R (net profit over money
actually at risk) and the running mean is tested against zero. Above the
configured minimum sample, if mean + sigma*SE < 0, new entries stop.
- SIGNIFICANTLY below, not merely below. A run of losers is ordinary variance
even for a profitable system; halting on the raw mean would be the same
act-on-noise error the MI gates exist to prevent. Using the standard error
means a wide spread simply demands more trades before the rule can fire.
- NET of swap and commission (ResolveClose already sums all three). Deliberate
and load-bearing: when the edge is zero, cost IS the expectancy, so a gross
version would measure a strategy nobody can trade.
- Reported in R so symbols, lot sizes and balances share one scale and one
mean. Trades without a stop are not scored rather than assigned a guessed R.
- LATCHED across restarts, like the daily halt and for the same reason: a
latch a reattach clears is not a latch. Clearing it means deleting the risk
state file, deliberately, after looking at why.
State is appended to the risk file length-guarded, so files written before this
still load and start their sample at zero rather than misreading.
Defaults 40 trades / 2 sigma; ExpectancyMinTrades = 0 disables it.
This does not make the strategy profitable and is not meant to. It stops paying
tuition on one the results say is losing, and does it on measurement rather than
on a drawdown limit finally being reached.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
b3b7e7bceb |
fix: excursion window must not depend on the barrier it sizes
DIRECTION IS NOT THERE, and this run is what establishes it. Three symbols:
raw ASYMMETRY clears on all three (p=0.0199 / 0.0050 / 0.0050)
norm ASYMMETRY collapses on all three (p=0.3433 / 0.5075 / 0.2736),
USDCAD landing BELOW its own null
RANGE control strengthens to 3-5x its null everywhere
Divide sigma out and the apparent directional signal vanishes entirely. What
cleared was volatility leaking through an unnormalised difference. Note this
would have passed any replication test: three instruments at p=0.005 is exactly
the evidence one would accept before committing to a rebuild, and the confound
reproduces perfectly. Replication was never going to catch it - only the
normalisation could.
Two defects of mine, both surfaced by the same run.
1. THE GEOMETRY DERIVATION WAS DIVERGING, NOT CONVERGING. It produced a
14.57*ATR stop and a 29.14*ATR target that only 5.7% of bars ever reach.
Excursions were measured over the barrier horizon; the horizon scales with
the target; the target is a quantile of the excursions - so target ->
horizon -> excursions -> target ran away, and "settled" only because the
horizon ladder caps at 384 bars. A saturated runaway, which the iteration
guard could not catch because it watches for OSCILLATION.
Fixed at the root: excursions now accumulate only over m_swingMedianBars -
the UNSCALED median ZigZag leg, a property of the instrument that owes
nothing to the barrier. The barrier walk still runs the full horizon,
because that is how long the trade is held; only the MEASUREMENT used to
size the barrier is confined to a geometry-independent window.
(The Min_Risk_Reward_Ratio warning fired correctly and is what flagged it -
the diagnostic worked while the derivation behind it did not.)
2. THE CONFOUND VERDICT WAS UNREACHABLE. `sizeCleared && !asymCleared` was
tested first and is true whenever size clears - i.e. always - so the branch
that NAMES the volatility confound never printed; all three symbols showed
the generic size-not-direction message instead. Verdict chain rewritten with
the specific case first, and the dangling elses my first patch introduced
removed.
FORCES A FULL RETRAIN (the excursion window changes every derived barrier).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
32ffeb99f3 |
fix: normalise the asymmetry target - the raw one is confounded by volatility
Three symbols ran the excursion test. RANGE/UP/DOWN cleared on all three;
raw ASYMMETRY cleared on EURUSD and USDCAD at p=0.0050 and not on SP500
(p=0.1045). That looked like the first directional signal this project has
found. It probably is not, and the test as built could not tell.
(up-dn) IS NOT SCALE-FREE. If sigma is predictable - and RANGE clears at ~4x its
null on every instrument - and the directional part is symmetric noise eps, then
up-dn ~ sigma*eps, so a large sigma pushes the value into BOTH outer terciles. A
pure volatility predictor scores positive MI against a 3-bin (up-dn) while
carrying no directional information at all. Crucially that confound REPLICATES,
so reproducing on two instruments is not evidence against it - and the effect
sizes fit it: asymmetry runs 1.3-1.6x its null where RANGE runs ~4x, and carries
~0.1% of the target's entropy against RANGE's ~0.9%. That is the shape of a
leaked fraction of the volatility signal, not an independent one.
So add (up-dn)/(up+dn): bounded in [-1,+1], volatility divided out, and the only
target a directional claim may rest on. The verdict now separates the cases and
NAMES the confound when raw clears while normalised does not, instead of
reporting the raw line as a finding.
Two bugs of mine in the same block, both caught by output rather than review:
- The derived-geometry line had a MISORDERED argument list: it printed
"stop 25.00*ATR (q3 of adverse travel)" - the quantile percentage as the
multiple and the multiple as the quantile. Real values were 2.61 stop /
8.03 target. A 25*ATR stop is absurd on its face, which is why it was seen.
- THE STOP QUANTILE WAS BACKWARDS, and this one changes labels. It was 0.25
"so ordinary noise does not reach it", but q25 means 75% of bars EXCEED the
stop - hit three times in four. The printed reachability said exactly that
("stop on 75.0% of bars"). Now 0.75. A quantile is a threshold, not a rate.
This is the entire reason reachability is measured and printed rather than
assumed.
Also raises BARRIER_DERIVE_MAX_PASSES 3 -> 5: SP500 did not settle in 3 (stop
still moving ~14% per pass) while EURUSD and USDCAD converged on pass 2. And
bounds both quantile indices with MathMin(..., n-1) so q=1.0 cannot run off the
end of the sorted array.
The geometry from the previous run is NOT usable and the asymmetry result is
unresolved, not established. Both are decided by the next run.
FORCES A FULL RETRAIN (the stop quantile changes every label).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
a7701f032b |
feat: derive the ATR multiples from measured excursions - no hardcoded geometry
The barrier was still two constants. SL_Mode/TP_Mode left the Inputs tab in |
||
|
AnimateDread
|
2c78f3b90d |
diag: is "optimal SL/TP" learnable? Score the features against excursions
Proposed direction: train the net to predict entry/SL/TP that maximise return
and minimise drawdown, rather than to classify direction. Before rebuilding a
head, measure whether the target is learnable at all.
That question splits into two that behave nothing alike:
HOW FAR price travels (MFE/MAE) - essentially volatility, and volatility
clustering is about the most robust regularity in markets.
WHICH WAY it goes first (the asymmetry) - direction, which is what every
noise-floor verdict in this project has been about.
Expectancy comes ONLY from the second. The first buys position sizing and
drawdown control - worth having under prop-firm limits, but not an edge: exit
management on RANDOM entries already moved the payoff ratio 0.92 -> 5.72 with
expectancy FLAT.
Crucially this is NOT already answered. Every MI figure here scored the
triple-barrier label, i.e. one specific question at one fixed geometry. A
noise-floor result there says nothing about whether excursion MAGNITUDE is
learnable - different target, different answer.
Four targets, and the verdict is the CONTRAST, printed explicitly because the
dangerous misreading of "UP clears" is "we can predict profitable trades":
RANGE (up+dn) - realised volatility, included as a POSITIVE CONTROL that
SHOULD clear. Every prior verdict here lacked a control
expected to pass; a range target at the floor indicts the
measurement, not the market.
UP / DOWN - MFE / MAE.
ASYMMETRY - up-dn, the only one that can pay.
Collected inside the walk the label already does (one max, one min per bar).
The early-out when both barriers resolved is GONE: it would have truncated the
excursions at whichever bar tripped the last barrier, making the measurement a
function of the CURRENT SL/TP - the circularity this is trying to escape. The
loop was already bounded by the horizon, so only the average cost moves.
Discretised into 3 EQUAL-FREQUENCY bins, so every downstream piece (block
permutation, null, p-value) is reused unchanged. Equal-frequency because MFE is
fat-tailed and fixed-width bins would put nearly every row in bin 0; it also
pins H(Y) at ln(3)=1.099 for all four, making them comparable to each other and
to the barrier label's ~1.02 instead of confounded by class balance.
Two bugs fixed in this code before it ever ran, both of which would have
produced a plausible quiet wrong answer rather than an error:
- TripleBarrierLabel early-returns on invalid ATR/close BEFORE the point the
accumulators were reset, so one bar's excursions would be cached under
another bar's index. Cleared at the top now, ahead of every return.
- An unresolvable bar is still flagged as labelled but carries excursions of
exactly 0. Under equal-frequency binning a block of identical zeros drags
the lowest cut onto zero and a third of the sample lands in one
uninformative bin - a depressed score that reads as "not predictable", a
false negative in the direction that would wrongly kill the idea. Rows
where both excursions are zero are dropped; price cannot travel zero both
ways over a whole horizon.
Read-only diagnostic. No topology or label change: no retrain of its own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
3482b6c238 |
feat: entry/SL/TP stop being inputs - the barrier geometry is measured
Three enums left the Inputs tab. They were three things a user had to pick and, in the tester, three more axes for a genetic optimization to overfit. Entry_Multiplier is pinned to MARKET. Its pending modes place the entry at a LEVEL while the rest of the pipeline measures from the bar open - the exact mismatch that manufactured the +0.097 R "retail fade" result later retracted as a fill artifact. This codebase's fill model cannot honestly simulate a pending entry, so it is no longer offered. SL_Mode/TP_Mode become a STARTING pair. ReportBarrierGeometryScan now ADOPTS its winner instead of printing "set SL_Mode/TP_Mode to X and retrain": - only when it clears the family-wise gate from |
||
|
AnimateDread
|
9e1c72aacc |
fix: make the indicator tuner actually measure, and gate what it installs
ROOT CAUSE of the zero spread measured on SP500 H1 2026-08-07 (all 17 candidates
returned exactly 0.00359 nats): the tune loop re-inits the indicators and then
scores, with no RefreshData() between.
ReInitADIndicators() does its part - Create() builds a NEW handle carrying the
new parameters, and the feature cache is flagged stale so features really are
recomputed. But BufferTempDataCompute() reads the CIndicatorBuffer objects, and
only Refresh() copies data out of a handle into those. So every candidate was
scored on values still held from the PREVIOUS handle. My earlier guess in the
diagnostic ("suspect the feature cache") was wrong: the cache invalidation works.
Two things land together, because neither is safe alone:
1. RefreshData() after the re-init, so a candidate is scored on its own features.
2. A SELECTION GATE on the install. bestScore is a MAXIMUM over candidates, and
the maximum of N draws from a null beats its incumbent almost every time - so
"it beat the incumbent" installs noise. This selector is the highest-stakes of
the three found in this audit because it ACTS: it overwrites the user's
configured indicator settings and forces BuildFreshTopology(), so the network
then trains on whatever the noise picked. Fixing (1) without (2) would have
made a dormant bug actively harmful.
The gate draws the winner's own permutation null once, then corrects the p-value
for having chosen it out of N with Sidak: p_family = 1 - (1-p)^N. Sidak rather
than the max-of-N resample used by the geometry scan because each candidate here
has a DIFFERENT feature set, so their draws cannot be pooled; Sidak needs only
the one null. Exact under independence, mildly anti-conservative under positive
dependence - stated in the comment rather than hidden. A rejected winner restores
the configured settings, which best[] cannot do since the descent mutates it.
Also reports the least-ready tunable handle's BarsCalculated(). IndicatorCreate()
calculates asynchronously, so if the spread is STILL zero the handles simply are
not done and the tuner needs to yield between candidates rather than score them
back to back - a state machine like the label prebuild. That distinction is now
readable from the log instead of requiring another guess.
No input, topology or label change: no retrain.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
e5ceed6466 |
fix: MI diagnostics never ran on a resumed model - the stated intent was never achieved
A comment above the diagnostic branch says it "runs even when the sweep does not: on a resumed model ... tying it to that gate meant the only way to see the answer on a running model was to delete the model." It does not. Moving the diagnostic out of the tuner's gate left it behind m_labelCachePrebuilt, which has the same effect: the eager label pre-scan runs only on a FRESH start, because a net loaded from disk labels lazily per bar. So on a resumed model the flag is false forever and the whole MI block - headline, positive control, alignment scan, lag profile, geometry scan, winner test, and the auto-tune line - silently never runs. Measured on SP500 H1 2026-08-07: attached at era 271, still nothing by era 314, zero MI lines in the day's log, and the only "label cache pre-built" entry predates the attach. It also explains the shape of every capture on 08-05/06: each one came directly after a weights reset. The situation the comment was written to eliminate is exactly the situation that persisted. So drive the pre-scan when it is the only thing missing. Safe on a trained net: its one fresh-net side effect, pushing the output-layer bias toward the dominant class, is already gated on m_eraCount == 0, and the advance gate in Train() sits ABOVE if(!m_trainRunActive), so the era loop keeps its state - training pauses for the scan (~1s at 38k bars) and continues from where it was, not from 0. Announced only on a start that actually armed, since StartLabelCachePrebuild() returns unarmed when history is not ready and is retried per bar event. NOT sampled from the lazily-filled cache instead: BuildMiSample skips bars with no cached label, so that would score whichever subset training happened to have visited - a biased subsample presented as a measurement, which is the failure this diagnostic exists to catch. Also corrects a claim in 0d58923's comment. It argued four consecutive "no improvement" runs were ~1-in-100,000 evidence the indicator tuner is inert, by multiplying 5.6% across four runs. They are not independent trials: the MI scorer is deterministic and all four covered nearly the same bars, so an incumbent that is the maximum on this data is the maximum on every run. One ~1-in-18 observation with three correlated repeats, ~5.6% - unremarkable. The same independence assumption that made the uncorrected lag profile star four lags. The candidate-spread line stands: it settles inert-vs-live directly. No input, topology or label change: no retrain. Training in flight stays valid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
0d5892357b |
diag: report the indicator tuner's candidate spread - "no improvement" is ambiguous
Auditing the other best-of-N scans after
|
||
|
AnimateDread
|
cccf94f9ca |
fix: correct the lag profile across lags too - it contradicted itself
|
||
|
AnimateDread
|
04ee2e113a |
fix: gate the barrier-geometry winner on a family-wise null, not its own
The scan ends by printing "set SL_Mode/TP_Mode to <winner> and retrain".
That advisory fired on `bestExcess > cfgExcess * 1.5` - a ratio between two
numbers, with no test that either is distinguishable from zero.
bestExcess is a MAXIMUM over the eligible candidates. The maximum of several
draws from a null sits well above any single draw from it, so a max-shaped
statistic tested against a single-candidate null crowns a winner on noise
almost every time. On SP500 H1 the winner is 2:8 at +0.00081 nats - and the
lag profile committed in
|
||
|
AnimateDread
|
3271f1ea93 |
diag: MI feature-lag profile - close the blind spot in every MI verdict so far
BuildMiSample samples features from ONE bar. So every "MI is at the noise floor" result this codebase has produced - including yesterday's p=0.18 on SP500 H1 - described the ENTRY BAR's 31 features only, while the network is fed 20 bars of them. If information lived at lag 7 and not lag 0, the report would have said "no signal" while the model could still learn. The diagnostic we have been making decisions on had a blind spot exactly the width of the input vector. Adds a FEATURE-side offset to BuildMiSample, which is not the same thing as the existing labelBarOffset and is not interchangeable with it. Shifting the LABEL changes which trade is predicted, so at any non-zero offset the features sit inside the labelled window and the score is lookahead - that is precisely what the alignment scan measures and correctly reports (4.7x more knowable 5 bars into a 128-bar window). Shifting the FEATURES keeps the label pinned to the entry bar, so every row stays causal. ReportFeatureLagProfile() then scores k = 0..historyBars against the same block-permutation null and reports the deepest lag that clears it - the lookback the data supports, versus the 20 that was picked by hand and never measured. The null is redrawn PER LAG: finite-sample MI bias moves with the realised class counts and bin occupancy, and different rows survive the validity checks at each lag, so one shared floor would be right for lag 0 and wrong everywhere else. Draw count is reduced accordingly (40, not 200) since cost is draws x historyBars; this figure decides a lookback, never a trade. MiShiftPad now also covers historyBars, keeping the fixed-pad invariant that makes two builds comparable row by row. Read-only - no input, topology or label change, so no retrain. Both builds 0/0. Build tag lag-profile-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
8ccbddb051 |
Add new research scripts for trading strategy analysis
- Implemented sqx_audit.py to audit StrategyQuant X trade lists, focusing on performance metrics and cost analysis. - Created sqx_portfolio.py to evaluate portfolio performance based on uncorrelated components and their impact on risk and return. - Developed swing.py to analyze cost ratios across different holding periods and assess swing trading structures. - Introduced test_management.py to investigate the effectiveness of exit rules on random entries and their impact on expectancy. |
||
|
AnimateDread
|
ceb6342dfd |
feat(ai): spread as a volatility-regime feature, and fix a stale-index cache in both new blocks
Adds spread/ATR and the spread change ratio as network inputs (EnableSpreadFeature,
default on). Spread is the one microstructure channel that is both FX-available and
genuinely historical in the Strategy Tester - "during testing, the spread is not modeled
but is taken from historical data" - so unlike swap, signed tick flow or depth of market it
is something a backtest can honestly validate.
What it encodes, stated precisely because the raw measurement overstates it.
research/test_spread.py found spr/atr the strongest single feature in this codebase, on 5
of 8 instrument/geometry cells at 2-4x any volume feature. But the barrier LABEL charges
the spread inside its own barriers, so a wide-spread bar is mechanically likelier to
resolve as a loss and the feature would partly be predicting its own cost model. Relabelling
at zero cost and re-measuring the identical feature showed 20-40% of it WAS that tautology
and the majority was not (XAUUSD retained 97%). What survives is a volatility-regime
reading: spread is near-fixed while ATR is not, so the ratio runs high exactly when
realised volatility is below its own ATR estimate, which genuinely predicts whether
ATR-scaled barriers get reached. It is UNSIGNED - Neutral-vs-directional only, never a side.
Also fixes a stale-index bug I introduced with the cross-asset panel and had just repeated
in the spread series. Both cached on length alone:
if(m_crossAsset.Bars() >= bars) return true;
MQL5 series indices are relative to NOW, so one new closed candle shifts every index by
one. Keyed only on length, the panel keeps serving its index 0 as a bar that is no longer
the newest, and every cross-asset value is read one bar out of step with the price features
sitting beside it in the same vector - silently, with no error and no shape change. This is
the same class of defect as the dtStudied watermark behind the zero-direction backtests.
Both now carry a datetime anchor on m_Time.GetData(0), the same invalidation key the
label/feature bar caches already use.
And a performance fix that fell out of it: with correct invalidation the panel rebuilds on
every new bar, and RefreshConvergedSignal runs per bar - which in the tester would mean one
full multi-symbol resample per simulated bar at training depth. Inference only reads bars
0..m_historyBars-1 plus the panel's own slow window, so it now requests exactly that. The
cache check is >=, so a deeper panel left from training still satisfies it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
8710240cd5 |
fix(signals): revive a dead MA model, and demote Sanyaku from state to event
Two defects surfaced by research/test_classic.py, both verified fixed by re-running the
transcription against 178k bars of EURUSD H1.
CSignalMA model 1 could never fire. For any recursive average - and MA_TYPE_EMA is the
shipped default - MA(i) = a*Close(i) + (1-a)*MA(i+1), so
DiffMA(i) = a * (Close(i) - MA(i+1))
DiffCloseMA(i) = (1-a) * (Close(i) - MA(i+1))
are positive multiples of one quantity and always share a sign. Model 1 asks for a close
BELOW a RISING average, which is precisely the combination that identity forbids: 0.000%
of bars, either direction, any symbol. The MQL5 standard library this was ported from
defaults to MODE_SMA, where the two are merely correlated - the bug arrived with the EMA
default, not with the port. Reading the slope one bar back (DiffMAPrev) breaks the tie for
every MA type while keeping the model's stated meaning. Now fires on 7.92% of bars.
CSignalIchimoku model 11 fired on 27% of bars at weight 100. Sanyaku is three standing
STATES conjoined with no transition term, so it held across long stretches - and being
last in the if-chain at the top weight, the module's highest-conviction reading was also
its most common one, overwriting all eight event models below it on a quarter of all bars.
The old comment rejected an event form because "demanding all three flip on the same bar
would fire almost never" - true, but that is not the alternative. Kouten is the TURN: the
ALIGNMENT transitions, and only one role need change for it to. Testing !Sanyaku(idx+1)
fires once per aligned stretch. Now 2.17%, in line with Kumo breakout (2.4%) and the
strong TK cross (1.1%). DataReady() extended one bar deeper to cover the lookback.
Neither pattern showed edge before or after; this is about the models meaning what they
say and the vote not being dominated by a constant.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
a77ff64b13 |
fix(deinit): a full model write was running ahead of the cheap cleanup
"Abnormal termination" is back, and this time it is not the arrows. The timing names the culprit exactly: 16:02:31.547 OnDeinit: shutting down 16:02:36.003 Abnormal termination <- 4.46 s, MetaTrader gave up 16:02:36.226 chart signals - persisted <- cleanup finished 0.2 s LATE OnDeinit called StopTraining() BEFORE the chart cleanup. StopTraining() finalises an in-flight run, and FinalizeTrainRun() restores the best checkpoint and then persists it - a full ~1MB model write per signal. So the expensive step ran ahead of the cheap bounded one, which is precisely the inversion the shutdown ordering exists to prevent. The previous fix put PersistWeightsOnShutdown last and missed that StopTraining smuggles a second save in at the front. Two changes: Cleanup now runs FIRST, then StopTraining, then the weight save. The visible teardown is cheap and bounded, so it always completes even when everything after it is killed. And the deploy-persist inside FinalizeTrainRun is suppressed during shutdown. RestoreWeights() is an in-MEMORY swap, so the best checkpoint is already the live net by that line, and PersistWeightsOnShutdown writes exactly those weights moments later. The old path wrote the same model twice per signal - eight full writes across four charts - for no benefit. A user-pressed Stop still persists immediately, because nothing else would. Compiles 0 errors / 0 warnings. Build tag deinit-order-v2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
7d038df749 |
research: export the feature matrix and a raw OHLCV grid for offline work
The bottleneck on this project has never been the modelling - it is that
every hypothesis costs a compile, a deploy, an attach and a log read, and
answers exactly one question. Days have gone into questions that are
seconds of arithmetic once the data is in hand.
Adds a RESEARCH-ONLY build, gated behind WARRIOR_EXPORT_FEATURES and
never compiled into a shipped binary, which writes two things to
Common\Files\Warrior_EA\Research\ and then does nothing at all:
<symbol>_<tf>_features.csv - one row per bar: index, time, OHLC, ATR,
and the m_neuronsCount feature values. Exactly what the network sees.
The raw bars ride along on purpose: with OHLC and ATR offline, every
barrier geometry, horizon and in-trade target is recomputable without
MetaTrader in the loop.
<symbol>_<tf>_rates.csv - raw OHLCV across a grid of 8 symbols x 5
timeframes. The 26 engineered features only exist for the attached
chart (indicator handles bind to PERIOD_CURRENT); raw rates do not, so
ONE attach yields the whole research grid. The bar time also makes
session/hour/day-of-week derivable - the only inputs in play that are
not a transform of the same OHLCV series.
Safety, because this binary gets attached to a chart on a LIVE ACCOUNT to
reach real history:
- OnTick returns immediately, so Expert.OnTick() - the entire trading
path - is unreachable regardless of the AlgoTrading toggle, the
signal state or the inputs. Structurally incapable of sending an
order, not merely unlikely to.
- No config lock. It never trains and never saves a model, so it has
nothing to protect against a concurrent chart - and taking the lock
would make it refuse to start exactly when the config it wants to
read is already open, which is when it is most useful.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
004f2a04f7 |
fix(diag): the symbol sweep was measuring its own sampling, not the market
Twelve cells came back with higher-timeframe "signal" 5-9x anything on H1, at p=0.005. It was an artifact, and the sweep's own columns gave it away: excess tracked the sampling STRIDE almost monotonically, and the three D1 cells - stride collapsed to 1-5 bars against a 128-bar horizon, i.e. ~99% window overlap - were the three highest. Three flaws, all the same family: comparing numbers without the spread that belongs to them. 1. THE NULL ASSUMED INDEPENDENCE THE LABELS DO NOT HAVE. Triple-barrier labels overlap; two rows less than one horizon apart share most of their outcome window. A free Fisher-Yates shuffle destroys that dependence along with the association, making the null far narrower than the truth and handing out significance that isn't there - Lopez de Prado ch. 4 arriving through the back door of the significance test. Now permutes contiguous BLOCKS of at least one horizon, so the null keeps the autocorrelation and the p-value means what it says. It degrades honestly: severe overlap leaves few blocks, the null widens, nothing reaches significance. The block count is now printed, because THAT - not the row count - is the sample size a p-value rests on, and a warning fires under 30 blocks so "not significant" is not misread as "no signal" when it means "not enough independent history to tell". 2. THE POSITIVE CONTROL'S STRENGTH DEPENDED ON THE DATASET. It paired each row's label with the NEXT SAMPLE ROW's, whose distance is the stride - so on M5, where stride ran 160-717 bars against a 128-bar horizon, it was pairing two windows that never overlap. All three M5 cells duly reported a FAILED estimator and voided their own results with nothing wrong. A control whose strength varies with the cell cannot certify the cell. Now pinned to a quarter of the horizon, where ~75% overlap is guaranteed by construction. 3. THE LOOKAHEAD VERDICT HAD NO MARGIN. It flagged 7 of 12 cells on gaps of 0.00008-0.00040 nats against a measured null sd of ~0.00030 - noise, every one. Now requires 3 sd, the same discipline the deploy floor applies to precision. Compiles 0 errors / 0 warnings, standard and Market. Build tag blockperm-v1. Supersedes every number from the sweep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
168422ff7a |
fix(labels): the 128-bar horizon ceiling was truncating the shipped label
The corrected geometry scan exposed something bigger than the geometry
question it was asked. Every pairing from 2:6 upward came back CLAMPED -
including 2:6, the SHIPPED configuration.
First-passage time for a driftless walk leaving [-m,+k] goes as m*k, and
the measured swing median here is ~12 bars at m*k=1, so 2:6 wants ~144
bars and 3:10 wants ~360. The ladder stopped at 128. A clamped label
stops meaning "does the target come before the stop" and quietly becomes
"...within 128 bars", while the deployed EA holds until SL or TP with no
bar limit. So the target the models have been trained on all along was
not the strategy the EA executes, and the trades it silently reclassified
as Neutral were the SLOW WINNERS - precisely the ones a 1:3 barrier
exists to capture. Timeout share stayed ~0% throughout, which is why this
never showed up: the truncation lands in Neutral, not in the timeout
counter that was watching for it.
Ladder extended to 384 (12..128, 192, 256, 384) so every selectable
geometry gets an honest horizon. Cost is one embargo of at most 384 bars
out of ~38k.
Second fix, same class of error as the H(Y) one: the scan's "best
eligible" was 2:2, a 1:1 barrier, against a shipped Min_Risk_Reward_Ratio
of 1:2. Training four topologies on that target would have produced a
model whose every setup is rejected at the door - the exact failure
behind four consecutive Market rejections for "no trading operations".
Sub-minRR geometries are now ineligible and marked [<minRR], printed
rather than hidden.
Also drops the dense-depth tag from the display name ("Perceptron 3L" ->
"Perceptron"). Depth is derived, so it names nothing a user chose; the
config tag [PAI-0be2] already disambiguates concurrent charts and does it
for every input rather than one. Full topology still logged by "config -".
Compiles 0 errors / 0 warnings, standard and Market. Build tag
horizon-384-v1. Changes the LABEL for every geometry, so the next scan
supersedes the previous numbers - and a retrain is required before any
model trained under the truncated target means anything.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
40af4a4b5b |
fix(labels): the geometry scan rewarded the labels it should reject
First run named 3:10 on all four charts, at 2.3x the configured 2:6. That answer was wrong and the fault was the ranking statistic. 3:10 wants a horizon of ~swingMedian*30 (~320 bars) and gets BARRIER_HORIZON_MAX. Clamped, most trades never resolve, the unresolved remainder all lands in Neutral, and H(Y) collapses. The old statistic divided the excess BY H(Y) - so a collapsing denominator made the most degenerate label look like the most predictable one. Every geometry from 2:6 upward was already showing the clamped h128, and the two widest scored highest, which is the fingerprint of the artefact rather than of signal. Two fixes: Rank on the raw excess in nats. Subtracting each geometry's OWN measured null already removes the class-balance bias, which is the only thing the normalisation was ever needed for. Disqualify clamped geometries outright rather than ranking them down. The deployed EA holds until SL or TP with no bar limit, so a truncated label trains the model on a question the strategy never asks. They are still printed, marked '!', so the disqualification is visible instead of a silent omission - and the scan now says so explicitly when nothing eligible is left, because "the limit is the feature set, not the target" is itself the finding in that case. The scan also reports each geometry's directional share and timeout share now. A label nobody can trade is not a candidate however well it scores, and that has to be visible in the same line as the score. Compiles 0 errors / 0 warnings. Build tag geometry-scan-v2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
f97ab9f1d6 |
feat(labels): measure which barrier is predictable at entry, don't guess
The alignment scan settled the shape of the problem: 4.7x more is knowable 5 bars into a 128-bar window than at the entry the model actually trades. A 6xATR target reached over 128 bars is decided overwhelmingly by what happens DURING the window, so whatever the entry state knows is buried under 128 bars of later noise. That is a property of the TARGET, and it is why four different architectures all landed on precision exactly equal to the base rate - no topology can undo it. So measure the target. For each SL/TP pairing a user can actually select, relabel the same sampled bars and score how much the SAME features say about THAT outcome at entry. Seconds, no training, no topology, and it runs on the diagnostic path that already exists. Ranked on excess over its OWN null as a share of its OWN H(Y), never on raw nats: each geometry has a different class balance, hence a different finite-sample bias and a different amount of information there to find, so raw MI would rank the most BALANCED label rather than the most PREDICTABLE one. The break-even win rate m/(m+k) is printed beside each so the ranking is read next to the bar the model must clear. Stated in the output because it is the easy thing to get wrong: chance precision EQUALS break-even at every geometry, so a tighter target does not hand you expectancy. It buys predictability - less noise piled on top of what the entry state knows - which is the one thing changing topology cannot do. Read-only by construction: it relabels a sampled copy via TripleBarrierLabel(), never writes the label cache (which belongs to the configured geometry), and restores the horizon and overrides it borrowed. The overrides apply only when BOTH are positive, so a half-set pair can never silently relabel a live run. Compiles 0 errors / 0 warnings, standard and Market. Build tag geometry-scan-v1. Redeploy only - no retrain to READ the ranking. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
4443ce85c1 |
fix(diag): the alignment scan cried misalignment at its own arithmetic
First run came back "WARNING - peak at k=+5, NOT 0 ... a feature/label
misalignment upstream of every topology". That was a false alarm produced
by the diagnostic's own design, and exactly the kind of plausible-looking
output this project has lost days to.
Bar indices are MQL5 SERIES indices - HIGHER index = OLDER bar
(TripleBarrierLabel walks its window as `for(t = idx-1; t >= idx-horizon;
t--)`, decreasing index = forward in time). The two directions therefore
mean opposite things and the scan treated them as symmetric:
k < 0 label belongs to a NEWER bar, its barrier window opens AFTER the
features exist. Nothing at bar i can legitimately know it, so a
peak here is real lookahead and a bug.
k > 0 label belongs to an OLDER bar, already k bars into its window by
the time bar i happens - so the features hold the realised first
k bars of that outcome. MI MUST rise with k. Arithmetic.
Only the k<0 side can indict the pipeline, and on the observed data it is
clean: -5/-3/-2/-1 all sit at or below the k=0 value and the noise floor,
so there is no lookahead - a real negative result, not an absence of
evidence.
The k>0 side is now reported as what it is, a second positive control,
with its gradient as the finding: 0.01881 at k=+5 against 0.00401 at k=0
means ~4.7x more is knowable 5 bars into a 128-bar window than at the
entry the model actually trades on.
Compiles 0 errors / 0 warnings. Build tag mi-align-v2. Redeploy only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
87c8656b53 |
diag(autotune): a positive control, and a scan that separates "no signal"
from "signal knocked out of step" Four architecturally different networks landed on the same precision - Buy 23-25% against a 25.4% base rate, Sell 19-22% against 22.0% - while making completely different calls (HYBRID votes Sell on 69% of bars, PAI on 41%). Precision equal to the base rate is what INDEPENDENCE looks like, and precision under independence is fixed by the label distribution, not by the architecture, so all four converging on it is arithmetic rather than coincidence. Accuracy meanwhile tracks coverage exactly as independence predicts (31.1/30.3/25.0 predicted vs 31.8/28.9/24.6 observed for PAI/CONV/HYB). But "no information in the data" and "information destroyed upstream of every topology" produce that identical picture, and the MI test alone cannot tell them apart either. Two additions: POSITIVE CONTROL. Three "measurements" in this codebase have turned out to be silent no-ops that produced plausible numbers - the MI scorer reading an array nobody filled, the eval-mode guard that switched off the imbalance correction, the alternation gate whose premise was never true. So the estimator now has to prove it responds to a signal known to be present before any floor reading is believed: the label of a neighbouring sample row, ~19 bars away and far inside the 128-bar barrier horizon, so the two outcome windows overlap heavily and MUST be associated. Same binning, same estimator. Near the floor => every MI figure is void. ALIGNMENT SCAN. Re-scores against the label taken from bar i+k for k in -5..+5. A peak at k != 0 is a feature/label misalignment - an off-by-one in the label index, a horizon applied to the wrong bar, a feature window that lags what it claims - which would destroy the information before any topology saw it and would look identical in every accuracy number this EA prints. A flat profile says the features simply do not carry this target. The sampled range is trimmed by |k| at both ends so a shift is measured rather than an edge effect, and both bars must carry a real label. Also: BuildMiSample publishes its stride instead of the report recomputing that arithmetic (it would drift), and the control sizes its buffers from its own sample count rather than the caller's. Compiles 0 errors / 0 warnings, standard and Market. Build tag mi-control-align-v1. Redeploy only - no retrain, no model deletion; the diagnostic runs on resumed models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
9a5f645dc3 |
diag(autotune): stop making the feature test cost a trained model
The permutation test lived inside TuneIndicatorsByFilter, which is gated on era 0 - correctly, because re-running the SWEEP would change the input vector out from under weights already fitted to the old one. But the test itself reads cached features and writes nothing, so none of that applies to it, and the gate meant the only way to see the answer on a running model was to delete the model. Today that price was PAI's 45 trained eras and CONV's 31, spent to re-ask a read-only question. Split into ReportFeatureLabelInformation(), called from the sweep when it runs and directly when it does not - a resumed model, a disabled tuner, nothing tunable. Once per attach either way. Compiles 0 errors / 0 warnings. Build tag permtest-v2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
9920754dec |
diag(autotune): five permutations was still a coin flip - use a real test
The 5-draw z-score shipped an hour ago disproved itself on its first run. All four charts scored the IDENTICAL 0.00401 nats on identical features and identical labels - and reported z of +1.3, +2.0, +4.0 and +4.7. Two "AT THE NOISE FLOOR", two "a real association", same data. The entire swing came from estimating the null's spread from five draws, where the standard deviation of the standard-deviation estimate is ~35%: the denominator was noisier than the effect it was judging. Replaced with an empirical permutation test. 200 draws, p counted by rank with the +1/(B+1) correction (Phipson & Smyth 2010) so p is never reported as exactly zero - no normality assumption and no spread to estimate. The strongest single column is tested against the null distribution OF THE MAXIMUM, which corrects for scoring 26 features at once by construction and is far less conservative than Bonferroni. Affordable because BuildMiSample is now split out of ScoreCurrentParamsByMI and runs ONCE for the whole test - every draw reuses that sample and costs a relabel plus 26 histogram passes, not 2000 feature extractions. The coordinate sweep still calls the combined form, which is correct there: each candidate changes the indicator settings, so its features really do have to be re-extracted. The verdict line keeps both questions apart and prints both answers: the p-value for "is it real", the excess as a percentage of H(Y) for "is it big enough to trade". At n=2000 those can disagree, and collapsing them into one word is how a worthless effect gets called a discovery. Compiles 0 errors / 0 warnings. Build tag permtest-v1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |