forked from mnbvc188199/Warrior_EA
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
AnimateDread
|
7414570d9d |
fix(diagnostics+calibration): the frozen-layer reading was a broken ruler; gate the operating point on a null of the maximum
Two defects behind the "training is highly unstable" report, from 101 eras of
SP500 H4 PAI logs. Neither was the optimizer.
1) LayerLearningReport's dW/W for BN layers divided by the WHOLE packed block.
getWeightsBN concatenates the outgoing dense matrix, gamma/beta, the running
mean/variance, the Adam moments AND BN_OPT_NX - the forward-pass scratch copy
of the normalized input. At era 101 bn1's dense matrix normed 15.1 against a
block norm of 15430.3, of which NX alone was 15429.3: the weights were 0.098%
of their own denominator, a 1022x inflation. NX is also near-constant between
era-end reports (same last forward pass), which pins the numerator down too,
so the layer read "bn1:0.000%" for 101 consecutive eras and was diagnosed as a
frozen first layer. It was the ruler that was broken. The ratio now covers
trainable parameters only (dense matrix + gamma + beta); mean/var/NX/Adam are
excluded. NX is reported separately because it is a health signal in its own
right - bn5 read nx 6.8e6 over 16 neurons, ~1.7e6 per unit against a healthy
~1.0, which is what a near-zero running variance in the denominator looks like.
NO historical dW/W reading on a bn* layer is admissible evidence that a layer
did or did not train. That includes every such claim in this repo's notes.
2) FitDirConfThreshold took a bare argmax of coverage x (precision - breakEven)
over 50 bins. Measured across 98 consecutive fits:
correlation(chosen threshold, win rate at it) = -0.056 over 0.00..0.74
win rate stdev across fits = 1.32pp
binomial SE of that win rate at ~1430 calls = 1.25pp
The correlation is zero - the margin does not rank trades - and the era-to-era
spread IS its own sampling error to within 0.07pp. So the objective was
coverage x (3.4 +/- 1.3) and the argmax over ~37 eligible bins returned
whichever bin drew the luckiest sample. The threshold teleported
0.42 -> 0.04 -> 0.74 in three eras, swinging OOS coverage 0% -> 39%, leaving
the era win rate measured on 1-5 calls and swinging 0% <-> 100%. That is the
entire reported instability.
The argmax is now adopted only if it beats a DETERMINISTIC fallback - the most
selective bin still clearing the coverage floor, chosen from the margin
distribution alone and never from a win rate - by more than a best-of-N
maximum could manage on noise, sqrt(2 ln N) standard errors. Same null-of-the-
maximum correction the deploy gate already applies to model selection.
A plain one-standard-error band was tried first and is NOT sufficient: its
edge is bestScore - bestSE, and with a 2.3pp edge against a 1.25pp SE that
edge is itself +/-50%, so the admitted set would still wander by half its own
width every era. The fallback has to be independent of the noisy quantity.
Simulated on the observed numbers: falls back every era at the current 2.3pp
edge (stable), adopts the argmax once a real edge reaches ~5pp.
NOT COMPILED - user compiles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
AnimateDread
|
444909d0a3 |
feat: S2 meta-labeling head - binary trade-quality model over the classic-candidate corpus
The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
ea9d86b3ee |
fix: dense backprop read the weight matrix transposed - on every backend
CaclHiddenGradient computed this layer's gradient as matrix_w[(outputs+1)*i + k] against a buffer whose actual layout (one row per NEXT-layer neuron, stride inputs+1) makes the correct read matrix_w[k*(inputs+1) + i]: the transpose for square layers, and for the non-square boundaries this EA actually builds (tapered stacks, the 3-neuron head) a mis-strided walk that ran past the buffer end - garbage on OpenCL, zeroed reads on the CPU DLL, so the tiers did not even agree with each other. Every gradient crossing a dense boundary on its way down - the entire learning signal reaching the BN/conv/LSTM front ends - passed through a fixed wrong matrix: feedback-alignment dynamics, not backprop, which is why nets still "learned something" and this survived. The book reference (NeuroNet_DNG) fixed this in a later article version; our kernel descended from the earlier one. Confounds every model-based negative verdict to date. Also in this commit, same root cause family: - per-sample UpdateWeightsAdam (OpenCL): input for slot group j was read at matrix_i[j] instead of matrix_i[j*4] (corrupted outer product past group 0), and dispatch dim 1 sized on ceil(inputs/4) left the bias column unreachable whenever inputs%4==0 - dense biases never trained on OpenCL. Rewritten as a lane-guarded scalar loop keeping our Adam conventions (sqrt-stored v, decoupled decay, both clamps, no sign gate). The batched accum path never had either bug; this kernel is what SetBatchSize(1) runs - including online continual learning on client machines, where OpenCL is the only tier. - conv backward passed raw (int)Activation() where the kernels expect NativeActivationCode(): NONE took the tanh branch (clamping a BN layer's unbounded z-scores), TANH took sigmoid, PRELU took none. Dormant only because the conv sits at layer 1 today. - hidden-gradient dispatch over Neurons()+1 dropped to Neurons(): biases get no backprop gradient and the extra work-item only ever read past matrix_o. All three backends (Network.cl, WarriorCPU.cpp, WarriorDML.cpp HLSL) changed in lockstep; DML gained an `inputs` constant to derive the row stride. New dense_backprop_check.cpp proves the CPU kernel is central-finite-difference consistent with the real forward kernel on 8x8, 64x3, 33x64, 5x3 (max diff 3e-9) and that all three activation branches match transcription. All 16 checks pass. Offline math check only - the in-situ proof remains the per-layer dW/W report on a real era. FORCES FULL RETRAIN. Both DLLs rebuilt and redeployed to MQL5\Libraries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
9a7c37f334 |
fix: live trades now use the geometry the gate certifies; perf: BN kernels
Three changes, one theme: the trade placed, the trade graded, and the trade computed are now the same trade. 1) GEOMETRY WIRE (correctness, the ranked #1 open issue). The measured barrier pair reached the LABELS only - OpenParams still placed orders at the enum geometry (2*ATR/6*ATR), so the deploy gate certified "reaches 1.62*ATR before 3.33*ATR above break-even" about trades the EA never placed. Published via g_DerivedSlAtrMult/g_DerivedTpAtrMult (ConfidenceBridge, same same-tick contract as the confidence globals, because OpenParams runs on the root signal which has no pointer to the AI filter). Two writers: DeriveBarrierGeometry at era 0, and the .cfg adoption a deployed model takes. Overrides both legs and both Intelligent modes - the certificate is exact or it is nothing. TP is ATR-anchored like the label, NOT risk-relative, so a floor-widened stop cannot reshape the certified target. 2) BATCH NORM RUNS DEVICE-SIDE ON OPENCL. Four kernels in Network.cl - forward, hidden gradient, gamma/beta accumulate, gamma/beta apply - each a line-for-line transcription of the host implementation (NormalizeHost / HiddenGradHost / StepGammaBeta) including every NaN guard, clamp, and the exact moment-write ordering. The host copies remain the runtime for the DLL and pure-MQL5 tiers and the reference the kernels must match. Because this box has no OpenCL platform, the safety story is layered: - shim validation: kernels compiled as C and driven against a fp64 host transcription over NaN-poisoned stats, NaN gamma, over-clamp inputs, the frozen path, both optimizers, 3 batches - ALL PASS, worst normalized diff 0.132 vs tolerance 1.0 - in-situ self-check: each kernel is compared against its host twin ON FIRST USE on the real device (SelfCheckBn*), covering what the shim cannot - arg indices and buffer bindings. Any disagreement resyncs from the good copy, latches all BN kernels off process-wide, and training continues host-side. A transcription bug costs a warning and some speed, never a poisoned .nnw. - sync discipline: BatchOptions is now a CBufferDouble with explicit authority tracking (m_bnDeviceAuthoritative). Checkpoints/saves pull read-only; restores/loads/resets push; a mid-batch handover drains the device gamma/beta accumulator into the host arrays so no sample is lost. 3) SMALL FIXES. Apply-kernel build failure now latches the dispatch path at init (one warning instead of warning + failed Execute). Build tag bumped to win-scoring-gpu-v1 - first tag change since expectancy-stop-v1 despite five binary-changing commits. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
899e0c66ca |
perf: mini-batch apply becomes a kernel - 8 weight-matrix transfers per batch become 0
Market builds cannot import a DLL, so OpenCL is the tier paying clients run.
It was several times slower than the CPU DLL, and the dominant reason was a
host-side optimizer step I shipped with the mini-batch work in
|
||
|
AnimateDread
|
371f8aaecd |
fix: the Adam second moment was never Adam - all four tiers
Root cause of the B=32 regression, and it predates F4 entirely. Every Adam
kernel stored v already square-rooted and then fed that stored value back in
as if it were the variance:
v_new = sqrt(b2 * v_old + (1 - b2) * g^2)
That recursion has a fixed point at v ~= b2 = 0.999 for ANY gradient below
unit scale, so the denominator stops tracking the gradient and Adam degrades
into plain SGD with lr = lt. Measured against the shipped WarriorCPU.dll
(batch_accum_check.cpp, TestOptimizerScaleInvariance), 4000 steps of a
constant gradient: 3285x less displacement at |g|=1e-5 than at |g|=1, where
a scale-invariant optimizer gives the same distance for both. After the fix
all six magnitudes read 1.199 and v tracks |g| exactly.
It hit conv/LSTM specifically because they sit behind a batch-norm with
running variance ~2.6e+05, so their gradients arrive divided by ~500 - deep
in the degraded regime - while the dense stack near the loss stayed in the
working one. In situ on SP500 H1: lstm1 dW/W 2.62/10.0/7.14% -> 0.024/0.022/
0.003%, conv1 decaying to 0.000% by era 30. NeuronBatchNorm.mqh already
squared v back for gamma/beta and its comment named the kernels as wrong,
which is exactly why gamma/beta kept training while the stages behind froze.
Persisted .nnw needs no migration - v keeps its std-dev meaning.
Also, the two ways F4 exposed it, both mine:
- No LR compensation for B fewer steps per era. sqrt(B) for adaptive methods
(Krizhevsky 2014; Granziol et al. 2022), applied once in
InitialEtaForOptimizer(). Linear scaling (Goyal et al. 2017) is for SGD.
- Plateau patience denominated in eras, so raising B made the ladder 32x more
impatient in its only unit. PAI converged at era 41 on ~49k updates where
the same config had been finding new bests at era 1028.
TrainPlateauPatienceEras() stretches it by the same sqrt(B).
TRAIN_BATCH_SIZE 32 -> 8 so the patience stretch stays affordable (8 -> 23
eras per stage, not 8 -> 45). Both helpers are identities at B=1.
Deploy gate: DEPLOY_MIN_SIDE_RECALL_PCT (10%) folded into tradeableOK. The
perceptron reported Sell:0% recall in all 41 eras, cleared the floor on Buy
alone at 36.6% vs 34% chance, deployed, and sprayed buy arrows. Folded into
the ranking key rather than checked at deploy time so a one-sided era cannot
become best-so-far in the first place.
Deinit: the arrow purge now runs BEFORE ExtPanel.Destroy(), an unbounded
CAppDialog teardown that sat ahead of it - the same ordering inversion the
rule there exists to prevent. CONV was force-terminated 4.8 s into OnDeinit
(vs ~1.1 s for the three that finished) having reached none of its cleanup,
so its arrows stayed on the chart. Steps are now timed in the log.
PurgeChart's verification rescan filtered on OBJ_ARROW, the same blind spot
as the bulk delete, so "persisted 10 ... cleared 0" passed silently. It now
walks every object type and reports the object counts when both are zero.
Both build variants compile 0 errors / 0 warnings; both DLLs rebuilt.
FORCES A RETRAIN (already forced by N1) and both DLLs must ship with the .ex5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
0c01dc279b |
feat: mini-batch gradient accumulation (F4), front-end-aware capacity budget (F6), split Wyckoff categoricals (N1)
Completes the 2026-08-09 training audit. FORCES A RETRAIN of every
Wyckoff-enabled config (N1 re-keys the fingerprint), and BOTH DLLs must be
redeployed alongside the .ex5 - they carry new exports.
F4 - mini-batch accumulation, TRAIN_BATCH_SIZE=32. Training was pure online
SGD (one weight update per bar), which is the mechanical source of the
era-to-era whipsaw every downstream guard was built to cope with. The O(n^2)
outer product is native - AccumulateWeightGrad / AccumulateWeightGradConv /
AccumulateBufferInto in Network.cl, WarriorCPU and WarriorDML - while the
optimizer step is host-side MQL5 shared by all tiers (ApplyAccumToBlock), so
there is one Adam/SGD implementation instead of four that can drift.
- the LSTM needs no outer-product kernel (WeightsGradient already holds the
sample's full dW) but could NOT simply be left un-zeroed between samples:
CPU_LSTMSeqBackward/DML_LSTMSeqBackward memset it on entry. Hence a
separate accumulator plus an elementwise add.
- batch-norm gamma/beta accumulate in host arrays, not new BatchOptions
slots - BN_OPT_STRIDE is baked into every persisted .nnw.
- scoped to pass 2; online learning keeps immediate updates. Every save /
checkpoint / scoring boundary flushes, scaling by the real sample count.
- degrades to per-sample updates (one log line) on a tier that cannot
accumulate, so old devices and DLL-free builds are unaffected.
- verified offline: DirectML/batch_accum_check.cpp drives the real exports
against an independent reference; at B=1 the accumulator matches the
shipped unbatched kernel's own gradient to 1.1e-16. Math only - the
in-situ check remains the per-layer dW/W report on a real era.
F6 - ComputeFirstLayerWidth budgeted against the RAW input width even where a
conv/LSTM front end had already reduced it, so an LSTM's dense stack was
charged for 1,280 inputs when it receives 64. Confirmed from the deployed
.cfg files: CONV, LSTM and HYBRID were all pinned at the 16-unit floor. Now
budgeted against the front-end output and capped at it (never fan out), with
the derivation reordered so both stages settle first.
N1 - EventCode/EventPhase/StructuralPhase are signed categoricals packing
direction and Wyckoff stage into one scalar across a sign discontinuity. Split
into direction + [0,1] magnitude, the same convention the base OHLC block uses.
Information-preserving; 13 readings now occupy 16 inputs.
Compiled clean (0 errors, 0 warnings); both DLLs rebuilt.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
AnimateDread
|
274630f802 |
fix: training-stability audit fixes F1/F2/F3/F5 - unbiased shuffle, real plateau escapes, fresh optimizer state on restore, pure OOS metric
Four of the six findings from research/training_pipeline_audit_2026-08-09.md (F4 mini-batching and F6 feature re-encode deliberately deferred - see the report's implementation-status section for why): - F1: pass-2 Fisher-Yates (and AutoTune's MI block shuffle) used MathRand()%, which is 15-bit - provably non-uniform on every full-history era over 32,768 queued samples. New 30-bit ShuffleRandomIndex(). - F2: plateau warm restarts were a no-op whenever eta already sat at its ceiling (the normal state of a non-regressing plateau) - the ladder was just a 24-era countdown. Restarts now overshoot to 5x the ceiling (PLATEAU_RESTART_BOOST) and anneal geometrically back over the patience window, SGDR-style; ETA_MIN widened 1e-4 -> 1e-5 so the decay schedule has real range. - F3: checkpoint restores put weights back but kept the rejected trajectory's Adam moments, so the optimizer immediately pushed back toward the rolled-back state (the restore->regress->restore oscillation). CNet::ResetOptimizerState() zeroes moments/momentum/step counters (weights, BN statistics, gamma/beta untouched) on every mid-run restore, every boosted restart, and the deploy-time restore that online learning continues from. - F5: batch-norm running statistics now freeze for the pass-3 OOS scoring walk, so the selection metric the checkpoint ranking and deploy gate read is a pure function of the checkpoint instead of partly measuring BN drift. Defensive unfreeze in FinalizeTrainRun covers stop-mid-pass; live/online adaptation and the OOS continual-learning simulation stay adaptive by design. Compiled clean (0 errors, 0 warnings) via the staged-tree recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
AnimateDread
|
f1b7dcf7f3 |
fix: correct MI sample alignment and improve BN weight diagnostic report
The MI sample builder used `MathAbs(labelBarOffset)` as a padding, causing rows from offset and non-offset builds to be paired with a double shift. This broke the positive control, failed the 5× gate, and voided all reported mutual‑information figures. Replace with the fixed `MiShiftPad` constant to ensure builds enumerate the same set of bars and row-k alignment is preserved. Add `BatchOptionsTotal()` to `CNeuronBatchNormOCL` and split the packed BN weight array in the learning report into separate norms for the outgoing dense matrix, gamma, beta, running statistics, and Adam moment buffers. This turns an ambiguous single‑norm reading into precise diagnostics that distinguish weight divergence from scaling issues. |
||
|
AnimateDread
|
5f647ba5db |
fix: improve error messages and suppress false sharing-violation logs
- BufferDouble: replace hardcoded "DirectML/CPU-DLL" with dynamic backend name and add buffer index/element count to all error prints for easier debugging. - NetPersistence: distinguish missing file from transient lock by probing FileIsExist before logging, eliminating false "sharing violation" warnings when no saved model exists on first run. |
||
|
AnimateDread
|
d7eea325fb |
refactor(ai): extract Layer.mqh and deduplicate AI config
- Moves CLayer neuron construction to AI/Impl/Layer.mqh to keep Network.mqh clean - Unifies four previously duplicated architecture initialisation blocks (MLP/CONV/LSTM/HYBRID) into a single shared function - Eliminates risk of behavioural drift where one architecture missed a setter, causing mismatched feature sets or targets |