A multi-chart comparison is only valid if every chart is identical except
the axis under test, and a drifted setting was previously invisible: the
model filename carries a HASH, so two charts that should match and do not
look merely "different" with no indication of which field moved.
Each signal now logs its effective config plus the raw fingerprint string,
unconditionally (not gated on VerboseMode). The six lines diff directly, so
an accidental divergence in study period, feature set, focal gamma or
anything else feeding training shows up at startup rather than as an
unexplained result hours later.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous note claimed the pooling stage was unfixable in the topology.
That is only true of TIME-axis pooling. The NeuroNet_DNG reference - whose
conv and pool kernels are byte-identical to ours - ties the pool to the
filter count (window = step = window_out), producing a clean
non-overlapping max-over-channels emitting one value per bar. So a correct
channel-pooling configuration does exist and needs no kernel change.
The real defect was that our window/step were never tied to window_out:
3/2 against 16 filters overlapped across the filter axis and straddled bar
boundaries.
Removal still stands, for a different and narrower reason: max-over-channels
at 16 filters reduces 320 conv outputs to 20 - one scalar per bar for a
420-wide input - and the first dense layer would fan OUT 20 -> 64 instead of
funnelling. The reference could afford that at window_out=4.
Comment-only. Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FeedForwardConv emits POSITION-MAJOR output, matrix_o[out + window_out * i],
so one bar's window_out filter responses are contiguous and consecutive bars
sit window_out apart. Both pooling implementations (FeedForwardProof and
CPU_FeedForwardProof) slide FLAT over that buffer - pos = i * step, reducing
`window` CONSECUTIVE elements. On a position-major layout those neighbours
are different FILTERS of the same bar, never one filter across time.
At the shipped 3/2 the pool computed max(bar0_f0, bar0_f1, bar0_f2), then
max(bar0_f2, bar0_f3, bar0_f4), with every 8th window straddling a bar
boundary. So it collapsed unrelated feature detectors into whichever fired
hardest, passed gradient to that winner only, and halved the feature map
while doing it - all below every learnable layer, where nothing above can
recover it. The removed inputs' own labels ("3 Bars") show time-axis pooling
was the intent throughout.
Measured cost: CONV sat pinned at ~40% balanced accuracy for 510 eras with
Sell recall 0%, while plain MLPs on the same data reached 57-61%. HYBRID,
which also carried this stage, came second-worst of the batch-norm group.
Not fixable in the topology: pooling one filter across time needs a stride
of window_out BETWEEN samples within a window, which a consecutive-window
kernel cannot express at any window/step. That needs a stride-aware kernel
in Network.cl + WarriorCPU.cpp + WarriorDML.cpp and a DLL rebuild, and is
only worth doing if a conv front-end earns its place without downsampling
first - with 20 sliding positions there is little to gain by halving them.
ConvPoolWindow/ConvPoolStep and their enums are removed with it, along with
the |CP: fingerprint term added earlier today.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Menon et al. 2021 (ICLR), "Long-tail learning via logit adjustment": add
tau*log(prior_c) to each class logit inside the training gradient. Softmax
CE on adjusted logits is consistent for BALANCED error - the metric
checkpoint selection already ranks on - so the loss and the deploy decision
finally optimize the same thing.
The engine already computed a true softmax + categorical-CE gradient and
wrote it over the per-neuron sigmoid delta, so this is an offset added to
three logits in the two places that gradient is built (backProp scalar path
and backPropOCL). No backend, kernel or DLL change; the forward pass and
every inference path are untouched, which is the point - the network learns
to absorb the offset, so its raw argmax becomes the balanced-optimal
decision with nothing applied at inference.
Replaces rather than stacks. Minority replay is disabled while this is on,
and the post-hoc inference prior is forced off. Stacking is not a
theoretical worry: simulated on the measured 1118/1119/34298 distribution
in the weak-signal regime, plain CE collapses to Neutral (33.4% balanced,
Buy 0%), replay reaches 48.1%, logit adjustment 50.9% with better balance -
and BOTH together score 45.4% with Neutral recall at 0%, worse than either
alone. Buda et al. 2018 predicts exactly that.
Motivation from the six-chart run: every topology took one direction to
~50% recall and abandoned the other, the direction chosen arbitrarily (the
batch-norm control went Buy 1% / Sell 42%, the inverse of the other five).
One era in 1,301 cleared the per-class recall floor.
Fingerprinted conditionally, so the converged 60.7% models on disk keep
their filenames and stay loadable as the fallback.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dividing a machine budget by the live chart count was wrong twice over.
The count is a snapshot taken when each net's pool is built, and charts
attach one at a time: five charts measured 10/6/5/4/4% of the same budget,
because the first only ever saw itself and the last saw all five. So the
earliest chart got several times the threads of the latest - skewing any
cross-topology comparison run on those charts, which is the exact thing
the setting existed to make fair. Nothing rebalanced afterwards either,
and rebalancing would mean tearing down a DLL context under a live trainer.
Both problems disappear once the answer stops depending on how many charts
are running. Each net now asks for a fixed 2 worker threads, converted to
the percentage the DLL wants from the detected core count.
Two is not a compromise: since the topology became data-derived the widest
dense layer is 64 units, so each ParallelFor has almost nothing to split
and per-dispatch overhead dominates. An MLP era cost ~66s at a wildly
oversubscribed 12 threads and ~80s at 1 thread - a 20% spread across a 12x
difference in thread count. Two per net also lands six concurrent charts
exactly on a 12-core box.
Removing the input costs nothing on the product side: a Market build has no
DLL tier at all, so it was already compiled out to a constant there and no
buyer could reach it.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The dense-depth tag separates MLP_3L from MLP_4L but not two charts that
differ by anything else - the batch-norm control is 3L on both sides, so
it put two identical "Perceptron 3L" streams in the log. Any config
difference at all changes the fingerprint by construction, so it is the
only discriminator that cannot go stale as inputs are added. The 4 hex
digits match the model filename's first 4, so a log line greps straight
to its .nnw.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Audited every input in Variables\Inputs.mqh against the filename hash.
Five changed the trained weights without changing the filename, so
switching any of them silently re-adopted a model trained under the old
value - the .cfg guard only catches it when the topology also differs, and
says nothing at all when it does not.
ConvPoolWindow / ConvPoolStep the Pool layer's window/step set how many
neurons it emits, resizing every dense
matrix above it
EnableMinorityReplay gates the replay loop and scales focal
gamma
OversampleParity sets the minority replica count
ConstrainReplay caps replicas and gamma
VolumeData tick vs real feeds different numbers into
the same input slot
PeriodMA / PeriodRSI seed the indicator tuner exactly as
MA_Type does - MA_Type was already hashed,
these two were not
Pool geometry is unconditional, matching how m_convFilterCount and
m_lstmHiddenSize are already treated. The replay knobs nest under
EnableMinorityReplay so turning replay off cannot re-key a model over a
parity value nothing reads. The three feature-value inputs are conditional
on the AI feature that consumes them, following the MACD/Ichimoku rule -
they also drive the classic MA/RSI votes, which are inference-only.
Re-keys existing models. Deliberate and free this cycle: the derived
first-layer width and the |BN: term already re-keyed everything, so this
is the cheapest moment it will ever cost.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MLP_3L and MLP_4L both identify as "Perceptron", so running them side by
side writes two interleaved streams of identically-prefixed lines and the
log cannot be split back apart afterwards - half a comparison run lost to
a naming collision rather than anything technical.
The dense layer count is exactly what AIType varies between them, so the
name now carries it: "Perceptron 3L", "Perceptron 4L", "Convolutional 2L".
Display only. m_id (the State\<id>\ folder) and the config fingerprint are
untouched, so no model file is re-keyed. Idempotent, because a failed init
leaves m_isInitialized false and this point can be reached twice on one
object.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two charts with the same AIType and the same retrain-affecting inputs
resolve to one .nnw/.cfg/.stats/checkpoint set. Both train and both save,
so whichever writes last wins and the other's eras vanish - silently,
because every individual file operation succeeds.
A five-chart comparison run hit this today: one chart was left at the
AIType default (HYBRID_2L), so two Hybrids shared State\HYB\...nnw and
the intended MLP_3L never ran. The only evidence anywhere in the log was
that model path appearing twice as often as the others.
The claim is a terminal-wide temporary global variable keyed on an FNV-1a
hash of the resolved filename - which is the correct lock identity, since
every retrain-affecting input is already folded into that name.
GlobalVariableTemp() gives an atomic create-if-absent, and a temporary
variable dies with the terminal, so a crash cannot leave a stale lock
blocking the next start. Within a session, a claim whose owning chart no
longer runs an expert is taken over; a chart reclaims its own entry across
a parameter change or recompile. Live charts only - tester and optimizer
agents are separate processes writing sandboxed _optcache copies, and
running one config across many agents is the point of an optimization.
Both builds compile 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The CPU-DLL thread pool was sized from TargetCPULoad undivided, on the
reasoning that only one pool is ever actively computing at a time. That is
true WITHIN a chart - MQL5 gives one chart's EA a single execution thread,
and every WarriorCPU.dll entry point blocks it until its ParallelFor()
completes, so the live net, the EMA shadow and HYBRID's fused pair take
turns. It does not hold ACROSS charts, which each get their own execution
thread and really do run their pools simultaneously.
At the 100% default on a 12-core box, five training charts asked for 12
threads each: 60 threads contending for 12 cores. Measured today, dropping
to ~2 threads apiece made every chart train "super fast". This had
previously been read as one architecture being mysteriously 10x slower than
another on the CPU-DLL tier while identical on OpenCL - oversubscription of
that degree degrades superlinearly and punishes whichever model issues the
most small sequential dispatches, which fits an MLP being the victim.
TargetCPULoad now means the budget for the whole machine, divided by the
number of charts running this EA. Counting charts is the correct axis:
concurrency here is one execution thread per chart, not one per CNet, and
the within-chart division that was previously removed stays removed.
Snapshot at pool-creation time on purpose - attaching another chart later
does not resize pools that already exist, because that would mean tearing
down a DLL context underneath a live training run. Skipped entirely in the
tester/optimizer, where the terminal already pins one strategy per agent.
This matters most for buyers: a Market build compiles the input out and
pins it to 100%, so they cannot reach the setting at all and would have hit
the pathological case with no way to diagnose or fix it.
The tier log line now reports the split rather than just the result.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deriving the first layer's width left NeuronsReduction and MinNeuronsCount
behind as inputs calibrated for something that no longer exists. Against a
hand-picked 500-wide first layer "keep 30%, floor at 20" produced a genuine
funnel - 500 -> 150 -> 45. Against the derived 64 it degenerates to
64 -> 20 -> 20: the reduction factor stops mattering after one step, and
"minimum neurons per layer" silently becomes the width of every layer but
the first. Two knobs whose labels no longer describe what they do.
The taper now runs geometrically from the derived first-layer width down to
a final hidden layer sized off the output count, spread evenly over however
many layers the chosen AIType implies:
MLP_3L 64 -> 28 -> 12 -> 3 29,151 dense weights
MLP_4L 64 -> 37 -> 21 -> 12 -> 3 30,450
CONV/LSTM/HYBRID_2L 64 -> 12 -> 3 27,763
and it stays a funnel at the floor, where the old rule could not:
D1 (first layer floored to 16) 16 -> 14 -> 12 -> 3
Both inputs are removed. With the width derived there is no freedom left in
the taper, so keeping either would only let the user contradict the
derivation. The layer COUNT stays selectable, because it is bundled into
AIType alongside the conv/LSTM front-end - depth is an architecture choice,
not a data-derived quantity, and pairing them means the two cannot
contradict each other.
m_minNeuronsCount / m_neuronsReduction survive as frozen members: nothing
reads them to build a topology any more, but they hold positional slots in
the .cfg sidecar and the weights fingerprint, and changing either value
would re-key every model on disk for no behavioural reason.
The DB config fingerprint drops both terms.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
InitialNeurons was an input whose only defensible value depends on two
things the user cannot see when picking from a dropdown: how wide the input
vector ended up after feature selection, and how much in-sample data the
study period actually yields. Left to a hand-picked constant it was badly
wrong - 500 units against a 420-wide input is 210,500 weights, 72% of a
292,583-weight model, against ~36,500 training bars of which only ~2,236
are directional. That is 6.6 weights per training bar, and it EXPANDS a set
of highly correlated inputs rather than compressing them.
The symptom was already in the logs and had been read as a depth problem:
the shallowest topology consistently beat the deepest (perceptron 52.7%
balanced, hybrid 41.3%). Over-parameterization predicts that ordering just
as well as covariate shift does, and only one of the two had been addressed.
ComputeFirstLayerWidth() budgets roughly one first-layer weight per
in-sample bar. Measured across the configurations in use:
M15 10y -> 256 units, 129,071 weights, 0.73 per bar
H1 10y -> 64 units, 28,727 weights, 0.65 per bar
H4 10y -> 16 units, 7,559 weights, 0.68 per bar
Two design points that matter:
- It estimates in-sample bars from the STUDY PERIOD and timeframe, not
from Bars(). What is downloaded grows over a terminal's lifetime, and a
topology that widened as history filled in would re-key its own weights
file and discard a trained model.
- The result is snapped down to a coarse power-of-two ladder, so the
estimate would have to be wrong by ~2x to change the answer.
Every field it reads is already part of the weights-filename fingerprint,
so the derived value needs no fingerprint entry of its own. The public
setter is removed - it could only have been called after construction, and
would either be ignored or silently re-key the model mid-run.
Where the data cannot support even the floor (D1 over 10 years is under
2,000 bars) it now says so and names the fixes, rather than quietly
training a model with more weights than examples.
The DB config fingerprint drops the term too, which re-keys existing
pattern databases once - correct, since a model an order of magnitude
smaller should not inherit the old one's win-rate history.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
With normalization enabled a forward pass is not a pure function of its
input - it also advances the running mean/variance. ValidateCpuInference
compares the live backend net against a throwaway pure-MQL5 clone loaded
from the just-saved .nnw, so its own reference pass left the live model one
EMA step ahead of the file the clone reads. The check would then have been
measuring its own side effect, and a marginal result decides whether
buyers' backtests are allowed to run DLL-free.
Adds CNet::SetBatchNormFrozen / CNeuronBatchNormOCL::SetStatsFrozen -
classic batch-norm inference semantics, statistics used but not updated -
and freezes both sides for the duration of the comparison. Not persisted:
it is a transient evaluation mode, not model state. Default stays
adaptive, which is what the rest of the system (online continual learning)
is built around.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The only bounded stage in the entire forward path was the sigmoid
classification head - every hidden stage is PRELU. That is a network with
no internal scale control, and the failure ordered exactly by depth: on
SP500 H1 the shallow perceptron held ~52% balanced accuracy while the
deepest topology sat on the 33.3% one-class floor, with the per-bar logit
spread decaying monotonically (0.45 -> 0.38 over ~200 eras) until the
evidence tilt fell under the class-prior tilt. That is the signature of
internal covariate shift, which chapter 6.1 of the reference book is
entirely about and which the NeuroNet_DNG engine addresses with a layer
this project never had.
Two mechanisms make this the right fix rather than more hyperparameter
nudging:
- it decouples WEIGHT_DECAY from the learned function (van Laarhoven
2017) - with a normalized layer downstream, decay can no longer grind
the discriminative signal away, it only rescales the effective
learning rate;
- it is the precondition for ever running an unbounded logit head here.
The 2026-07-27 attempt blew up (IS error 5.6e15) precisely because
nothing upstream constrained scale.
Implementation notes:
- CNeuronBatchNormOCL computes host-side rather than as a fourth copy of
a kernel across Network.cl + WarriorCPU.cpp + WarriorDML.cpp. The math
is elementwise O(n); this way it behaves identically on all four
compute tiers, needs no DLL rebuild, and cannot drift between
backends. Same precedent as the softmax+CCE gradient and the
per-sample loss weighting, both computed in MQL5 for that reason.
- Statistics are exponential moving, not a stored mini-batch: training
is pure online SGD, one update per sample, so there is no batch to
average over. BatchNormWindow is an EMA window length.
- gamma/beta are excluded from weight decay, deliberately - decaying
gamma toward zero is the exact pathology being fixed.
- The layer self-sizes from whatever sits below it, because a conv/pool
stage's output width is derived inside the CNet constructor and is not
knowable to the topology builder.
- Checkpoint capture/restore/blend carry gamma/beta and the running
statistics alongside the dense matrix, so the plateau ladder cannot
restore a mismatched pair.
- SeedOutputLayerBias accepted only an exact defNeuronBaseOCL as the
weight-carrying penultimate layer; with normalization enabled that is
the batch-norm layer, so the cold-start bias seed would have silently
stopped being applied.
- Refuses to build, loudly, if a topology asks for normalization with no
compute backend at all - rather than quietly training a different
architecture than the one requested.
EnableBatchNorm (default on) and BatchNormWindow (1000 samples) are
inputs so the effect can be A/B'd without a recompile. Both feed the
weights-filename fingerprint, appended conditionally so existing non-BN
configs keep their fingerprints and are not forced to retrain.
Verified: analytic gradients match finite differences to 1.5e-7 relative
over 200 random cases; a faithful port of the full forward/backward chain
collapses to the 33.3% floor by era 4 without this layer and holds
36-43% with it. Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A .nnw persists the ARCHITECTURE, not just the weights: Save writes
(int)activation per neuron and Load reads it straight back. The activation
chosen in BuildFreshTopology() therefore only ever reached a brand-new
topology - every reload restored the file's value and the next save wrote it
back out, so a wrong value could never heal while the source read as though
it were already fixed.
That is how five models kept training with an unbounded NONE classification
head for a full day after the 07-28 revert to SIGMOID. Confirmed by parsing
the binaries: 848cb42c.nnw / 2e754b43.nnw carry `act=NONE` on the 3-neuron
output layer, while a genuinely reset model of the same config carries
act=SIGMOID. In the log it showed as negative "OOS raw out" values -
impossible under sigmoid - escalating to a 4.14e13 logit spread with all
three classes numerically identical (input-independent output) and balanced
accuracy pinned on the 33.3% one-class floor.
- OutputLayerActivation() is now the single source of truth, called by both
BuildFreshTopology() and the new load-time repair, so the two can no
longer diverge the way a duplicated literal did.
- CNet::EnforceOutputActivation() re-asserts it after Load and reports the
stale value; CExpertSignalAIBase::EnforceTopologyContract() logs the
repair loudly, since weights learned under the old head may not be worth
keeping even once the head is corrected.
- Hidden layers are deliberately left alone: they legitimately differ per
stage (PRELU dense/conv, NONE pool, TANH LSTM).
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Arrow cleanup existed on two paths - the panel's reset-weights, and the
topology-mismatch discard - but both are gated on there being a saved .nnw to
delete. The third case had no cleanup at all: a fresh topology at era 0 with no
weights behind it, which is what a changed config produces. A new fingerprint
makes a new m_fileName, so the previous model's files are not "discarded", they
are simply not this model's files, and nothing ever cleared the chart.
That is not cosmetic. Arrows outlive the model that drew them twice over:
1. The chart objects live in the CHART, not the sidecar, so they survive a
remove/re-add, a recompile, a restart and a fresh deploy no matter what
happens to any file on disk.
2. SaveChartSignals() rebuilds the sidecar by SCANNING the chart for
SIG_ARROW_PREFIX objects. So the first save of the fresh run adopts the
dead model's calls and writes them out under the NEW model's filename -
laundering them into the new model's history where nothing can separate
them afterwards.
Extracted the duplicated cleanup into ClearPersistedChartSignals(reason) - it
cancels the deferred restore queue, deletes m_fileName + ".arrows", clears the
namespaced chart objects and logs why - and called it from all three paths.
The call sits at the BuildFreshTopology() call site, not inside it: the genetic
tuner rebuilds a throwaway topology per candidate (AutoTune.mqh) and must never
touch the chart. All three sites run after m_fileName has its config fingerprint
appended, so they target the right sidecar.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Evidence (MQL5\Logs, SP500 H1, 2026-07-29):
Perceptron era 61 Buy 32% Sell 27% Neut 94% bal 51%
LSTM era 160 Buy 16% Sell 11% Neut 98% bal 42% (peaked 49% @ era 44)
Hybrid era 179 Buy 5% Sell 2% Neut 99% bal 35% (peaked 41%)
CONV era 228 Buy 2% Sell 4% Neut 99% bal 35% (peaked 40% @ era 122)
Every model peaks early then decays monotonically toward Neutral, and nothing
stops it: the restore-best-weights + decay-eta handler is gated on
m_bestPassedRecall, which stays false forever when no checkpoint ever clears the
per-class floor. CONV ran 228 eras with eta pinned at its 0.000300 start. The
plateau ladder cannot end such a run either (stage 3 refuses to deploy without a
recall pass, so it resets ~27 times), making it a 1000-era one-way trip.
The gate's own justification had expired. It was written when the pre-pass
tiebreak was blended-accuracy-only, where "best" really did mean "called Neutral
most confidently". The balanced-selection change replaced that with
`balancedOosEra > m_bestBalancedOos` plus an isFullyCollapsedEra exclusion, so a
Neutral-only era now scores ~33% - the FLOOR of the balanced metric - and cannot
anchor the checkpoint at all. Pre-pass "best" now means "most class-balanced so
far", which is worth defending; and isWorseEra is itself a balanced-accuracy
regression, so it cannot fire merely for trading Neutral calls for Buy/Sell.
The original concern still holds while the best-so-far IS near-collapse, so the
escape is margin-guarded: defend the checkpoint only once balanced accuracy sits
more than BALANCED_WORTH_DEFENDING_MARGIN_PCT (5pp) above the one-class floor of
100/3. Against the run above that engages for all three stuck topologies
(42.3/41.3/50.0 vs a 38.3 threshold) while a genuinely collapsed run still
explores freely.
Two inputs restored to the regime that actually produced a deploy:
- MinRecall 60 -> 40. The one successful auto-deploy in the logs (Hybrid, 28th
00:50, best balanced 66.0%) ran against a 40% floor. 60 has never been shown
reachable here - a floor above what the config can reach is the same "target
set too high" failure the surrounding comment already warns about.
- OversampleParity 60 -> 90. 60 overcorrected. Runs now START Neutral-dominant
(Buy 0-11% recall at era 1) and call Buy/Sell on 0-4% of bars against a ~6%
true base rate - under-calling, with no headroom to converge down from. The
deploying run began at Buy 90% / Sell 36%, 24% of bars called, and settled into
the floor from above. Raw over-calling is the intended starting condition; live
calls are base-rate-calibrated by AILogitPriorStrength, which is why the input's
own note says to judge over-calling by live-fired precision, not raw counts.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The simple panel showed "Buy/Sell accuracy: IS x% OOS y%" from m_cumIs*/
m_cumOos*, which are monotonic lifetime counters: never reset per era (only on
reset-weights) and restored from .stats across restarts. So the number is the
average over EVERY era ever trained. At era 217 one more era moves it by well
under a percent - it reads flat whether training is healthy or dead, and a model
that started badly and has since recovered still shows low.
That is the only number the non-verbose panel offered, so there was no way to
tell "still improving" from "stuck" while watching four charts.
Added the actual gate. The plateau ladder only auto-deploys a checkpoint that
cleared m_minDirectionalRecallPct on EVERY class (Buy AND Sell AND Neutral,
default MinRecall=60%). If nothing ever clears it, stage 3 deliberately refuses
to deploy, resets the ladder and keeps training to the era cap - correct
anti-collapse behaviour, but externally indistinguishable from being stuck.
Panel now shows:
- era against the cap, not just the era number
- the accuracy line explicitly labelled "(lifetime avg)"
- best balanced accuracy vs the per-class floor it must clear
- eras since best + ladder stage, so plateau escapes are visible
Display only - no training, selection or convergence logic touched.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
REFACTOR_NOTES.md records what was found, what was changed, what was
deliberately left alone, and the one investigation that is still open (the
MLP CPU-DLL slowdown, with the parameter counts that rule out my earlier
"largest weight matrix" explanation).
Also restores the missing opening rule on ReInitADIndicators' comment banner.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ExpertSignalAIBase.mqh was 8216 lines: the class declaration followed by 87
method bodies covering training, labelling, feature extraction, persistence,
chart drawing, online learning, the GA auto-tuner and inference, all in one
file. Train() alone is 1492 lines; a change to arrow drawing meant scrolling
past the era loop.
Moved the bodies into Expert\AIBase\, included at the bottom of the original
after the class declaration:
Training.mqh 1607 era loop, plateau ladder, checkpoint select, deploy
Features.mqh 1093 indicator creation + per-bar input feature vector
ChartUI.mqh 634 arrows, arrow persistence, status panel, cleanup
Persistence.mqh 492 .stats/.cfg sidecars, CPU-inference validation, copy
OnlineLearning.mqh 461 live continual learning, EMA shadow, OOS simulator
Labels.mqh 309 ZigZag pivot labels, async label-cache prebuild
AutoTune.mqh 275 genetic tuner (population, crossover, halving)
Inference.mqh 235 softmax, prior calibration, class priors
ExpertSignalAIBase.mqh 8216 -> 3131 (declaration + topology build only)
This is a pure relocation - verified mechanically, not by eye: HEAD's file
reconstructed from the eight partials plus the surviving remainder is
byte-identical to HEAD, span for span (scratchpad verify_split.py). No
declaration moved, no signature changed, no code rewritten, so behaviour is
unchanged by construction.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DRY - topology construction
---------------------------
CSignalCONV and CSignalHYBRID each built the Conv+Pool front-end from scratch;
CSignalLSTM and CSignalHYBRID each built the LSTM stage from scratch. The
duplicates had already drifted: HYBRID guarded the LSTM step with
MathMax(1, historyBars/2), CSignalLSTM divided unguarded, so a historyBars of 1
gave two different steps for what is documented as the same layer.
Extracted AddConvPoolStage() and AddLstmStage() onto CExpertSignalAIBase. The
three overrides are now compositions:
CONV = AddConvPoolStage
LSTM = AddLstmStage
HYBRID = AddConvPoolStage && AddLstmStage
HYBRID's "matches the standalone CONV front-end exactly, then adds LSTM" is
enforced by construction instead of by comment. Took the guarded step for both.
Also fixed a descriptor leak the duplicates shared: on a failed topology.Add()
the CLayerDescription was neither owned by the array nor deleted.
Dead code
---------
- CNet::SaveCheckpoint / CNet::LoadCheckpoint (123 lines). Superseded by the
in-memory CaptureWeights/RestoreWeights pair; Network.mqh:1312 already said so
("This replaces the file-based SaveCheckpoint/LoadCheckpoint"). Zero call
sites - every remaining mention was a comment. The five comments that
referenced them have been reworded rather than left dangling.
- CExpertSignalCustom::CheckForDuplicateTrade / FindLastTradeIndex /
UpdateTradeStatusAndExit: declared, never defined anywhere, never called.
They only made it look as though duplicate-trade detection existed.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FileOpen(FILE_WRITE) truncates its target on open. CNet::Save already staged
the .nnw through a temp file + rename for that reason, but the three sidecars
written beside it did not:
.stats ExpertSignalAIBase.mqh:5918
.arrows ExpertSignalAIBase.mqh:6224
.cfg ExpertSignalAIBase.mqh:7329
Two defects followed.
1. An interrupted write published a truncated sidecar. For .cfg that is the
worst case: LoadAndCompareTopologyConfiguration() reads a short file as a
mismatch, which discards the trained model and restarts from era 0.
2. Windows file sharing is a mutual contract - a writer opened with no
FILE_SHARE_* blocks every concurrent open regardless of the reader's flags.
All three read paths carry FILE_SHARE_READ|FILE_SHARE_WRITE specifically so
a tester agent can read them while a live chart runs; an exclusive writer on
the same path defeated that.
Extracted CNet::Save's proven pattern into System\AtomicFile.mqh
(AtomicWriteBegin/AtomicWriteEnd) and routed all four writers through it. This
also encodes the FileMove gotcha once instead of per call site: the destination
location comes from FILE_COMMON inside the 4th arg, NOT inherited from the
source, and getting it wrong moves the file to the wrong sandbox silently.
Also fixed while in these functions:
- SaveTopologyConfiguration had 13 copy-pasted 6-line error blocks that each
returned WITHOUT FileClose(handle), leaking the handle on every write
failure. Collapsed to one ok-chain that closes exactly once. The on-disk
field order and types are unchanged (asserted during the rewrite) so existing
.cfg files still load.
- SaveChartSignals documented that pruning runs only after a successful write
("a failed write above leaves both the file AND the chart untouched") but
never checked any write result, so a partial write still deleted the chart
objects. Results are checked now, making the existing comment true.
Compiles 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The DFA (Direct Feedback Alignment) option was never a correct implementation:
it deterministically flipped the sign of half of all gradients based on
connection index parity, causing permanent gradient ascent for those weights
and guaranteed divergence. The backward pass was also incompatible with the
OpenCL/DirectML neuron model (layer.Total() == 1). This change removes all DFA
logic, including the enum value and `DfaFeedbackSignal` method, and replaces it
with plain gradient descent in all momentum update kernels. The `optimizer`
kernel argument is retained for binary compatibility but is no longer used.
The DFA optimizer was never real Direct Feedback Alignment — it flipped
gradient signs for half of each weight tensor deterministically, causing
permanent gradient ascent and divergence. Its backward pass was also
incompatible with the OpenCL/DirectML layer model. Remove the enum
entry, all related logic in the OpenCL kernel weight update functions,
and the `DfaFeedbackSignal` method. The `optimizer` kernel argument is
retained (unused) to preserve binary compatibility with existing DLLs.
Add type checks to correctly cast `CNeuronBaseOCL` objects in gradient clipping,
softmax evaluation, and `backProp`/`backPropDfa` loops. Previously, all neurons
were assumed to be `CNeuronBase`, causing invalid pointer casts and incorrect
gradient/output access for OpenCL layers. This ensures proper support for both
CPU and OCL neuron implementations.
Replace dynamic_cast with explicit type checks using obj.Type() and static casts in CaptureOptimizerSnapshot, RestoreOptimizerSnapshot, and SetOptimizerForAllNeurons. This avoids the runtime overhead of dynamic_cast inside critical loops, improving performance when capturing/restoring optimizer snapshots and setting optimizer for all neurons.
Changed updateInputWeights methods to accept pointer arguments by value (e.g., CLayer* instead of CLayer*&) to prevent unintended pointer reassignment and improve safety. Added overloads in CNeuronBase and CNeuronBaseOCL to support both generic CObject* and derived type pointers, unifying the virtual interface and resolving inconsistencies in network weight update logic.
Changed updateInputWeights signature from accepting `CObject*` to `CObject*&` to allow the method to modify the caller's pointer, preventing potential object copying or pointer invalidation. A large block of outdated commentary in ExpertSignalAIBase.mqh was also removed, cleaning up documentation no longer applicable after previous refactoring.
Introduce Direct Feedback Alignment (DFA) backward pass with gradient clipping, feedback matrix initialization, and a dedicated backPropDfa method. Add optimizer snapshot/restore hooks (CaptureOptimizerSnapshot, RestoreOptimizerSnapshot, SetOptimizerForAllNeurons) to temporarily switch the entire network's optimizer for replay-only updates during pass 2, preserving the original optimizer state. Support all neuron types including dropout, deconv, LSTM, and softmax in the snapshot logic.
Introduce an `optimizer` parameter to UpdateWeightsMomentum, UpdateWeightsConvMomentum, and UpdateWeightsAdam kernels. When set to a non-zero value, the gradient used for weight updates is multiplied by ±1 based on the parity of the weight index, implementing a basic feedback alignment signal for experimentation. When zero, the standard gradient is used unchanged. This allows A/B testing of alternative learning signals without modifying the rest of the training pipeline.
Cleaned up the HTML across all language versions by removing unnecessary div and br tags, restructuring headings, and fixing table formatting. Aligned the structural flow and terminology between the Chinese, Turkish, and English descriptions, corrected translation inconsistencies (e.g., mixed-language table headers in Turkish), and unified the listing of features and workflows for a more professional and consistent presentation on the marketplace.
Eliminate the separate `AI_TOPOLOGY_PRESET` enum and input.
Fold the topology presets directly into `AI_CHOICE` as new combined values (MLP_3L, MLP_4L, CONV_2L, LSTM_2L, HYBRID_2L) plus `AI_NONE`.
Remove the `TopologyPreset` input variable and update default `AIType` assignments.
Update the market description to reflect the simplified single‑selector interface.
**Why:**
Users previously had to choose an AI architecture and a topology preset separately.
Now the UI shows one coherent selector that bundles architecture with its appropriate dense‑layer depth, reducing complexity and preventing mismatches.
Allow multiple AI models to be active simultaneously by switching from
a single AIType selection to individual boolean Enable flags. Also
simplify build tag by removing date prefix.
In AI/Network.mqh, return early from InitDirectML during
tester/optimization/forward runs to prevent agent-side file-lock
failures caused by rapid stop/restart cycles accessing DLL imports.
In Expert/ExpertSignalAIBase.mqh, add MathIsValidNumber checks in
CalibratedConfidenceMagnitude and SignaledConfidence to safely handle
NaN values, and refactor ShutdownChartCleanup to accept a preserve
flag, avoiding unnecessary chart purges during tester runs for faster
shutdowns. Also add m_purgeChartOnDestruct member.
In AI/NeuronDirectML.mqh, clean up a minor comment formatting issue.
In inference-only backtests, dtStudied could be ahead of the test range, causing new-bar detection to freeze. Replaced with m_lastBarTime to keep detection aligned with runtime history. Added diagnostic logging when a non-neutral softmax output is neutralized by prior correction. Also added validation for order_price, sl, and tp in stop-checking functions to catch non-finite or negative values.
Added a new `m_modelLoadedFromDisk` flag to distinguish models loaded from a saved `.nnw` file from fresh random topologies. Modified the readiness gates in `LongCondition` and `ShortCondition` so that an inference-only tester run can execute trades using a model loaded from disk even if its persisted `trainingComplete` flag is `false`. This enables replay of a partially trained exported model without requiring full convergence, while still blocking fresh random-weight topologies from trading.
Introduce a global boolean `g_signalsVisible` to control whether signal
objects are displayed across all timeframes or hidden entirely. When
enabled, chart arrows and restore objects are set to `OBJ_ALL_PERIODS`;
otherwise they use `OBJ_NO_PERIODS`, allowing signals to be shown or
hidden at runtime without losing saved state.
Add `ShouldTraceTradeRejections()` helper that returns true only when
running in the Strategy Tester, optimization, or forward testing modes.
Use it to print diagnostic messages when trades are rejected due to a
prohibition signal or when `OpenLongParams`/`OpenShortParams` fail to
produce valid stop/take-profit levels. This provides targeted debugging
output without cluttering live trading logs.
Wrap the `TCWarnIfSlow` calls in `CExpertCustom::OnTick` and the
`TCWarnIfMemoryAbove` call in `CExpertCustom::OnTimer` with `VerboseMode`
guards. This silences routine diagnostic noise during normal operation
while still allowing detailed performance tracing when `VerboseMode`
is enabled.
