bfc1da9de1
BuildFeatureWindow() replaces eight hand-rolled copies of the same loop
and feeds the window OLDEST BAR FIRST. Every copy fed it newest-first,
because MQL5 timeseries indices run backwards and `r + b` with b ascending
walks into the past.
Harmless for PAI and CONV - a dense layer learns a weight per position
either way, a conv learns time-mirrored kernels. Not harmless for the
recurrent stacks:
- LSTM_SeqStepForward reads `inputs + t*Iw`, so step t is block t.
- It writes output[] only when t == steps-1: the visible output IS the
last hidden state.
- c_t = f*c_{t-1} + i*g decays toward the start of the sequence.
lstm_seq_flowcheck.cpp measured block 0's influence on the output at
1.2e-2 of block T-1's, at the shipped forget bias of 1.0.
So the bar being PREDICTED sat at the far end of the decay and the output
was handed to the OLDEST bar in the window - the exact inverse of what the
window is for. ~80x backwards on LSTM and HYBRID, on all three tiers
(OpenCL kernel, CPU DLL, pure-MQL5 inference), which is why it never
surfaced as a backend discrepancy.
This does not create edge - the MI diagnostics read at the noise floor
(p=0.4975) with a working positive control. It makes the one hypothesis
those diagnostics explicitly do NOT cover testable: they are marginal and
per-bar, and state they "cannot rule out one that only exists in
combination or across time". The sequence model is the instrument for
across-time structure and it has been crippled, so that hypothesis has
never been honestly tested.
Fingerprint gets an unconditional |WIN:2 - the vector keeps its shape and
its features, so a stale .nnw would load cleanly and run a model fitted to
one ordering against the other, silently. Re-keying every config is the
point, not collateral damage. FORCES A FULL RETRAIN.
Also: the now-relative bar caches are re-keyed on the two live paths.
EnsureBarCachesCapacity() was only ever called from training paths, but
once m_trainingComplete is set ScheduleTrainingIfNeeded() routes every bar
to RefreshConvergedSignal() and Train() is never re-entered - so nothing
cleared the feature cache again for the life of the process. A chart that
trained to convergence kept replaying the rows computed for the last
training era's bar grid: the live signal froze at its convergence-time
value, and OnlineLearnStep() backpropped those stale features against
freshly resolved labels. Backtests were never affected (an inference-only
process never allocates the arrays, so every read recomputes).
Compiles clean: 0 errors, 0 warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1294 lines
88 KiB
MQL5
1294 lines
88 KiB
MQL5
//+------------------------------------------------------------------+
|
|
//| Topology.mqh |
|
|
//| |
|
|
//| Network bootstrap and topology construction: the derived shape |
|
|
//| (width/taper/depth/conv filters/LSTM hidden), the conv, LSTM and |
|
|
//| batch-norm stages, BuildFreshTopology and InitIndicators. |
|
|
//| |
|
|
//| PARTIAL IMPLEMENTATION FILE - not standalone. |
|
|
//| CExpertSignalAIBase method BODIES only. The class declaration |
|
|
//| lives in Expert\ExpertSignalAIBase.mqh, which includes this file |
|
|
//| at the bottom, after the declaration. Do not include it |
|
|
//| anywhere else and do not compile it on its own. |
|
|
//+------------------------------------------------------------------+
|
|
#ifndef WARRIOR_AIBASE_TOPOLOGY_MQH
|
|
#define WARRIOR_AIBASE_TOPOLOGY_MQH
|
|
//+------------------------------------------------------------------+
|
|
//| Common network bootstrap shared by every AI signal: sets up |
|
|
//| indicators, then loads a saved network or builds a fresh one |
|
|
//| whose only per-signal-type difference is AddCustomLayers(). |
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::InitNeuralNetwork(CIndicators *indicators)
|
|
{
|
|
if(m_isInitialized)
|
|
return true;
|
|
if(indicators == NULL)
|
|
return false;
|
|
m_indicatorsPtr = indicators;
|
|
if(!CExpertSignalCustom::InitIndicators(indicators))
|
|
return false;
|
|
if(!CExpertSignalAIBase::InitIndicators(indicators))
|
|
return false;
|
|
Net = new CNet(NULL);
|
|
if(CheckPointer(Net) == POINTER_INVALID)
|
|
return false;
|
|
//--- Size the first dense layer to the data. HERE and only here: it must be settled before the
|
|
//--- fingerprint below (which hashes it) and must never move afterwards - see ComputeFirstLayerWidth()
|
|
//--- and the note on fingerprint-feeding members at the top of this file. InitIndicators() above is
|
|
//--- what finalises m_neuronsCount, so this is the earliest point the input width is actually known.
|
|
m_initialNeuronsCount = ComputeFirstLayerWidth();
|
|
//--- Same rule, same place, same reason: the conv and LSTM stages are also sized from the data rather
|
|
//--- than configured, and both feed the fingerprint below, so they have to settle here too.
|
|
//--- Unconditional - a plain MLP simply never builds the stages these describe, and branching on the
|
|
//--- topology type would make the fingerprint depend on which subclass is asking.
|
|
m_convFilterCount = ComputeConvFilterCount();
|
|
m_lstmHiddenSize = ComputeLstmHiddenSize();
|
|
//--- Depth LAST of the four: it is derived from the first-layer width above, so it cannot be settled
|
|
//--- before that one is. All four are overwritten from the .cfg further below if this configuration
|
|
//--- already has a trained model - see the adopt-don't-compare block there.
|
|
m_hiddenLayersCount = ComputeHiddenLayerCount();
|
|
//--- The name used to carry a dense-depth tag ("Perceptron 3L"), from when AIType let a user pick
|
|
//--- MLP_3L vs MLP_4L and the depth was the only thing separating two charts of the same family.
|
|
//--- Depth is DERIVED now (see ComputeHiddenLayerCount), so it names nothing anyone chose - it is an
|
|
//--- internal shape detail leaking into a product surface a customer reads. Dropped: the config tag
|
|
//--- appended below ([PAI-0be2]) already disambiguates concurrent charts, and does it correctly for
|
|
//--- every input rather than just this one. The full topology is still logged once at startup by the
|
|
//--- "config -" line, which is where that detail belongs.
|
|
//--- Per-configuration fingerprint appended to the weights filename so that every distinct
|
|
//--- combination of RETRAIN-AFFECTING inputs gets its OWN persistent .nnw/.cfg, instead of all
|
|
//--- combinations sharing one file keyed only on symbol/period/output/optimizer. This is what lets a
|
|
//--- genetic/complete optimization that sweeps network params (neuron counts, layers, reduction,
|
|
//--- history bars, study period, feature set, focal gamma, OOS split, recall/WR targets, ...) build
|
|
//--- each combo's model exactly ONCE and then reuse it on every later pass that revisits that combo -
|
|
//--- previously each differing pass overwrote the single shared cache and retrained from scratch, so
|
|
//--- there was no cross-combination reuse at all. The topology .cfg check further below still runs as
|
|
//--- a secondary guard (and catches a rare hash collision by mismatching and retraining).
|
|
//--- Deliberately covers ONLY params that change the trained weights. Inference-only gates
|
|
//--- (Min_Vote_Open's confidence floor, SignalClusterWindow) and post-training/live settings (money
|
|
//--- management, trailing, entry, filters) are excluded, so changing those still reuses the exact
|
|
//--- same model - matching the pre-existing behavior the optimizer already relied on.
|
|
//--- 2026-08-01: SL/TP LEFT that exempt list. They used to be pure execution settings; the
|
|
//--- triple-barrier relabel makes them the barriers the TARGET is defined by (see TripleBarrierLabel),
|
|
//--- so changing either now changes every label and therefore every weight. A model trained at
|
|
//--- 1:3 must never be silently reused at 1:1. This is the rule from the fingerprint audit applied
|
|
//--- to the newest weight-affecting inputs: what shapes the labels shapes the hash.
|
|
//--- 2026-07-30: every DERIVED value left this hash - first-layer width, dense depth, conv filters,
|
|
//--- LSTM hidden size, and the retired reduction/minNeurons pair. They were legitimately here while
|
|
//--- they were functions of hashed INPUTS, which made them redundant-but-harmless. They stopped being
|
|
//--- that when the capacity budget started measuring the symbol's real bar count: a filename keyed on
|
|
//--- a measured quantity changes the moment more history downloads, so the EA would look for a file
|
|
//--- that does not exist, start from era 0, and orphan a fully-trained model - silently, since a
|
|
//--- missing cache is the normal first-run state and logs as such. The derived shape is pinned in the
|
|
//--- .cfg instead (see the adopt-don't-compare block in LoadAndCompareTopologyConfiguration), which is
|
|
//--- the correct home for it: it describes the model that EXISTS, not the config that asked for it.
|
|
//--- m_studyPeriod is gone for a simpler reason - the input it mirrored no longer exists.
|
|
string fp = StringFormat("%d|%d|%d|%d|%d|%d|%.2f|%d|%d|%d|%d|%d|%d|%d|%d|%d|%d",
|
|
m_optimizationAlgo, m_historyBars, m_outputNeuronsCount,
|
|
m_neuronsCount, m_minTrainYear, LEGACY_CONVERGE_WR_SLOT, m_fractalPeriods,
|
|
//--- LEGACY SLOT (was m_focalGamma, removed 2026-07-31). It was a double fed
|
|
//--- to a %d conversion, so it always contributed the literal below rather
|
|
//--- than the configured gamma - the shipped fingerprints read ...|40|0|30|...
|
|
//--- Writing the same literal keeps every existing model's filename intact.
|
|
m_minDirectionalRecallPct, 0, m_oosSplitPct, m_swingConfirmationBars,
|
|
(int)m_useVolumes, (int)m_useTime, (int)m_useATR, (int)m_useSwingContext,
|
|
(int)m_useNews, m_newsFeatureWindowMinutes);
|
|
//--- The one derived value that DOES belong here, and only when it is not derived at all: a forced
|
|
//--- depth is a developer override (see ForceHiddenLayers), so a build that pins one must not adopt
|
|
//--- the .cfg of a build that derived it. Conditional, so the shipping value of 0 leaves the hash
|
|
//--- exactly as it reads above.
|
|
if(ForceHiddenLayers > 0)
|
|
fp += StringFormat("|FHL:%d", ForceHiddenLayers);
|
|
//--- THE TRIPLE-BARRIER SHAPE LEFT THIS HASH ON 2026-08-07, when SL_Mode/TP_Mode stopped being inputs
|
|
//--- and became MEASURED by ReportBarrierGeometryScan. It is the same rule that moved the horizon out
|
|
//--- (and the derived topology values before it, see above): a filename keyed on a measured quantity
|
|
//--- changes the moment the measurement does - more history downloads, a few more bars shift which
|
|
//--- pairing wins - and the EA then looks for a file that does not exist, starts from era 0, and
|
|
//--- orphans a fully-trained model silently. Measured values are PINNED IN THE .cfg instead, which is
|
|
//--- read back and adopted on load, so a trained model keeps the geometry it was actually trained on.
|
|
//--- Nothing replaces it here on purpose: the .cfg is the record, and re-measuring never happens for a
|
|
//--- model that already exists.
|
|
//--- MA/RSI + AD feature flags appended separately to keep each StringFormat call's arg list modest.
|
|
//--- m_useMA/m_useRSI belong here for the same reason every other feature flag does: they change the
|
|
//--- input-vector width (see InitIndicators()'s m_neuronsCount += 5/+1), so a model trained with them
|
|
//--- on must never silently reuse a cache trained with them off. m_neuronsCount alone (listed above)
|
|
//--- captured the WIDTH but not the composition, so two different feature sets summing to the same
|
|
//--- width could have collided onto one cache file - these two flags close that gap.
|
|
fp += StringFormat("|%d|%d|%d|%d|%d|%d|%d|%d",
|
|
(int)m_useMA, (int)m_useRSI,
|
|
(int)m_useADCumulativeDelta, (int)m_useADShorteningOfThrust,
|
|
(int)m_useADWyckoffEventStream, (int)m_useADWyckoffFailedStructure,
|
|
(int)m_useADWyckoffSignificantBarInversion,
|
|
//--- starting MA TYPE (MA_Type input): changes the MA feature's values, so a change
|
|
//--- must invalidate the cache. The auto-tuned type/period themselves live in the
|
|
//--- .nnw indicator-param block (Flatten/Unflatten), not here - this is the seed only.
|
|
(int)MA_Type);
|
|
//--- MACD/Ichimoku feature flags, appended ONLY WHEN ENABLED rather than unconditionally like every
|
|
//--- flag above. Both spellings are equally correct as a fingerprint (deterministic either way, and a
|
|
//--- model with these on can never collide with one that has them off), but appending them
|
|
//--- unconditionally would have changed the hash of EVERY existing config the moment this feature
|
|
//--- shipped - re-keying and forcing a full retrain of already-converged models that don't use MACD or
|
|
//--- Ichimoku at all. Conditional append leaves those fingerprints byte-identical. The seed periods go
|
|
//--- in for the same reason MA_Type does above: they change the feature's values. Anything added here
|
|
//--- in future should follow the same rule.
|
|
if(m_useMACD)
|
|
fp += StringFormat("|MACD:%d:%d:%d", (int)MACD_PeriodFast, (int)MACD_PeriodSlow, (int)MACD_PeriodSignal);
|
|
if(m_useIchimoku)
|
|
fp += StringFormat("|ICHI:%d:%d:%d", (int)Ichimoku_PeriodTenkan, (int)Ichimoku_PeriodKijun, (int)Ichimoku_PeriodSenkou);
|
|
//--- Cross-asset panel: conditional append, per the rule above, so existing fingerprints are untouched.
|
|
//--- ONLY the flag and the feature count go in. The panel's actual composition - which reference pairs
|
|
//--- were found in Market Watch, and therefore which currencies it can index - is a MEASURED property
|
|
//--- of the terminal, exactly like the bar count the header warns about. Keying the filename on it
|
|
//--- would orphan a fully-trained model the moment the user adds or removes a Market Watch symbol,
|
|
//--- silently, since a missing cache reads as a normal first run. The composition is logged at build
|
|
//--- time and pinned in the .cfg instead.
|
|
if(m_useCrossAsset)
|
|
fp += StringFormat("|XA:%d", CROSSASSET_FEATURES);
|
|
//--- Spread feature: conditional append, same rule. Nothing measured goes in - the spread series
|
|
//--- itself is market data, not configuration.
|
|
if(m_useSpreadFeature)
|
|
fp += "|SPR:2";
|
|
//--- Batch normalization changes the LAYER COUNT, not just the weights, so a model trained with it
|
|
//--- must never load into a topology built without it (and vice versa) - the .cfg guard would catch
|
|
//--- the mismatch and retrain, but only after a confusing failure. Appended conditionally, following
|
|
//--- the same rule as MACD/Ichimoku above: a config with batch norm off keeps the fingerprint it
|
|
//--- already had, so shipping this does not re-key and force a retrain of every existing model.
|
|
if(EnableBatchNorm && BatchNormWindow > 1)
|
|
fp += StringFormat("|BN:%d", BatchNormWindow);
|
|
//--- Changes the training gradient, so a model trained with it must never load into a run
|
|
//--- without it. Conditional append, same rule as MACD/Ichimoku/BN above: a config with
|
|
//--- this OFF keeps the fingerprint it already had, so the already-converged models on
|
|
//--- disk stay untouched and remain loadable as the fallback if this regresses.
|
|
if(m_logitAdjustTau > 0.0)
|
|
fp += StringFormat("|LA:%d", (int)MathRound(m_logitAdjustTau * 100.0));
|
|
//--- 2026-07-29 audit of every input in Variables\Inputs.mqh against this hash. Five were changing the
|
|
//--- trained weights without changing the filename, so switching any of them re-adopted a model trained
|
|
//--- under the OLD value - the exact trap that the .nnw architecture incident already cost a day to
|
|
//--- (see EnforceTopologyContract): the .cfg guard would eventually mismatch and retrain, but only
|
|
//--- after a confusing failure, and a matching topology would not mismatch at all.
|
|
//--- LEGACY SLOT. The oversampling/replay inputs this encoded were removed 2026-07-31 (see the
|
|
//--- class-imbalance block in Variables\Inputs.mqh); the replay path itself is gone. The literal is
|
|
//--- the exact string the shipped defaults produced - EnableMinorityReplay=true, OversampleParity=90,
|
|
//--- ConstrainReplay=true - so every model already on disk keeps its filename and stays loadable.
|
|
//--- Dropping the segment instead would re-key EVERY model and force a from-scratch retrain of the
|
|
//--- one topology currently converged and trading, which is a steep price for cosmetics in a hash
|
|
//--- nobody reads. Same treatment as LEGACY_CONVERGE_WR_SLOT / LEGACY_STUDY_PERIOD_SLOT.
|
|
fp += "|MR:1:90:1";
|
|
//--- Feature-value inputs, each conditional on the feature that reads it actually being on - the same
|
|
//--- rule the MACD/Ichimoku blocks above follow. Tick vs real volume feeds different numbers into the
|
|
//--- same input slot (Features.mqh's m_Volumes.Create), and PeriodMA/PeriodRSI seed the tuner
|
|
//--- (ADIndicatorTuner.mqh) exactly as MA_Type does - MA_Type was already hashed, these two were not.
|
|
//--- All three also feed the CLASSIC MA/RSI votes, which are inference-only; gating on the AI feature
|
|
//--- flag is what keeps a classic-signal tweak from re-keying a model that never saw it.
|
|
if(m_useVolumes)
|
|
fp += StringFormat("|VOL:%d", (int)VolumeData);
|
|
if(m_useMA)
|
|
fp += StringFormat("|MAP:%d", (int)PeriodMA);
|
|
if(m_useRSI)
|
|
fp += StringFormat("|RSIP:%d", (int)PeriodRSI);
|
|
//--- INPUT WINDOW ORDER. Appended UNCONDITIONALLY, which is a deliberate break from the "conditional
|
|
//--- append so existing fingerprints stay byte-identical" rule every block above follows - and the
|
|
//--- reason is exactly why that rule exists in the first place. Every model on disk was trained on a
|
|
//--- window fed NEWEST-BAR-FIRST; BuildFeatureWindow() now feeds it oldest-first (see its definition
|
|
//--- comment for the LSTM measurement that forced it). The vector has the same SHAPE and the same
|
|
//--- features, so nothing downstream would fail: a stale .nnw would load cleanly, pass the .cfg guard,
|
|
//--- and run a model fitted to one input ordering against the other - silently, forever. That is the
|
|
//--- precise failure this hash exists to make impossible, so here re-keying every config is the
|
|
//--- CORRECT outcome, not collateral damage. Version it rather than toggling a flag: if the ordering
|
|
//--- is ever revisited, bump the number instead of trying to reconstruct which models predate what.
|
|
fp += "|WIN:2";
|
|
//--- FNV-1a 32-bit -> 8 hex chars: compact, deterministic, order-stable, collision-safe enough for
|
|
//--- the small optimizer grids in play (a collision would merely fail the .cfg guard and retrain).
|
|
uint fpHash = 2166136261;
|
|
int fpLen = StringLen(fp);
|
|
for(int fpi = 0; fpi < fpLen; fpi++)
|
|
{
|
|
fpHash ^= (uint)StringGetCharacter(fp, fpi);
|
|
fpHash *= 16777619;
|
|
}
|
|
m_fileName += "_" + DoubleToString(MathRound(m_outputNeuronsCount)) + "_" + DoubleToString(MathRound(m_optimizationAlgo)) + "_" + StringFormat("%08x", fpHash);
|
|
//--- Finish the display name with the model's short id and the leading 4 hex digits of that same
|
|
//--- fingerprint, so every log line and panel names the model file it belongs to. The dense-depth tag
|
|
//--- added at the top of this function separates MLP_3L from MLP_4L, but NOT two charts that differ by
|
|
//--- anything else - the batch-norm control was 3L on both sides, which put two identical
|
|
//--- "Perceptron 3L" streams in the log the first time this was tried. Any config difference at all
|
|
//--- changes the hash, by construction, so it is the discriminator that cannot go stale as inputs are
|
|
//--- added - but it is NOT unique on its own. The fingerprint deliberately omits the topology TYPE,
|
|
//--- because the file path already separates it (State\CONV\ vs State\LSTM\ vs State\HYB\) and hashing
|
|
//--- a value that is constant within a folder would add no discriminating power while re-keying every
|
|
//--- trained model on disk into a forced retrain. The consequence is that CONV, LSTM and HYBRID at the
|
|
//--- same depth with the same inputs hash IDENTICALLY - a 2026-07-30 deploy came back with three charts
|
|
//--- all tagged [4109]. Their files were never at risk; the TAG was simply unable to do its one job.
|
|
//--- Prefixing m_id restores uniqueness on the display side without touching m_fileName, and the hex
|
|
//--- half still greps straight to the .nnw inside the folder the prefix names.
|
|
string cfgTag = " [" + m_id + "-" + StringSubstr(StringFormat("%08x", fpHash), 0, 4) + "]";
|
|
if(StringFind(ID, cfgTag) < 0)
|
|
ID += cfgTag;
|
|
//--- One self-verifying config line per chart, deliberately NOT gated on VerboseMode. A multi-chart
|
|
//--- comparison is only valid if every chart is identical except the axis under test, and until now
|
|
//--- a drifted setting was invisible: the filename carries a HASH, so two charts that should match
|
|
//--- and do not look merely "different" with no indication of WHICH field moved. Printing the raw
|
|
//--- fingerprint string makes the six lines directly diffable - any accidental divergence in study
|
|
//--- period, feature set, focal gamma or anything else that feeds training shows up as a textual
|
|
//--- difference at startup instead of an unexplained result three hours later.
|
|
Print(ID + ": config - " + IntegerToString(m_hiddenLayersCount) + " dense from " +
|
|
IntegerToString(m_initialNeuronsCount) + " units | batchnorm " +
|
|
((EnableBatchNorm && BatchNormWindow > 1) ? "ON(" + IntegerToString(BatchNormWindow) + ")" : "OFF") +
|
|
//--- "requested", not the bare number: the EFFECTIVE tau is capped against the head's usable
|
|
//--- logit range and cannot be known until the class priors are measured, so printing 1.00 here
|
|
//--- read as the value in force when every chart was actually running 0.35. The real figure is
|
|
//--- logged once per run by ApplyLogitAdjustment().
|
|
//--- The ONE class-imbalance mechanism. There is deliberately no "| replay ON/OFF" beside it any
|
|
//--- more: that field reported a path which had already been dead for the whole shipped
|
|
//--- configuration, which is exactly the kind of line that makes a log look informative while
|
|
//--- describing nothing (see the class-imbalance audit in Variables\Inputs.mqh).
|
|
" | class-imbalance " + (m_logitAdjustTau > 0.0
|
|
? "logit-adjust(tau " + DoubleToString(m_logitAdjustTau, 2) + " requested)" : "OFF") +
|
|
" | input " + IntegerToString((int)m_historyBars * m_neuronsCount) +
|
|
" (" + IntegerToString((int)m_historyBars) + " bars x " + IntegerToString(m_neuronsCount) + ")" +
|
|
//--- The front-end stages are DERIVED (see ComputeConvFilterCount/ComputeLstmHiddenSize), so
|
|
//--- without them this "self-verifying" line verified only half the topology - it printed the
|
|
//--- dense taper while the conv/recurrent stages that actually dominate CONV/LSTM/HYBRID were
|
|
//--- invisible. Shows the width flowing INTO each stage as well as out of it, because the
|
|
//--- interesting failure is a stage that expands rather than compresses.
|
|
FrontEndConfigSummary());
|
|
//--- Kept as its own line and deliberately free of any per-chart prefix INSIDE the string, so the six
|
|
//--- startup lines diff textually against each other. The model file goes on the line below rather than
|
|
//--- here for the same reason: it necessarily differs per topology (it carries the State\<id>\ folder),
|
|
//--- so folding it in would make every fingerprint line differ and destroy the diff.
|
|
Print(ID + ": fingerprint - " + fp);
|
|
//--- The resolved path is DebuggingMode-only: the tag above already names the folder (its m_id half)
|
|
//--- and the file's hash suffix (its hex half), so this line is derivable rather than new information,
|
|
//--- and a third startup line per chart is not worth spending on a user who will never open the file.
|
|
if(DebuggingMode)
|
|
Print(ID + ": model file - " + m_fileName + ".nnw");
|
|
//--- Strategy Tester / optimizer: target a LOCAL (agent-sandboxed, non-FILE_COMMON) cache file
|
|
//--- instead of the shared production weights, so genetic/complete optimization passes on this
|
|
//--- same agent can reuse an already-trained model whenever the topology-relevant inputs
|
|
//--- (neuron counts, layers, history bars, output count, opt algo, study period, ...) are
|
|
//--- unchanged from a previous pass, instead of re-running every training era from scratch each
|
|
//--- pass. The live/manual-chart production .nnw/.cfg under FILE_COMMON are never touched by
|
|
//--- this path, so a backtest can never corrupt or overwrite the deployed live model.
|
|
bool inTesterOrOpt = MQLInfoInteger(MQL_TESTER) || MQLInfoInteger(MQL_OPTIMIZATION) || MQLInfoInteger(MQL_FORWARD);
|
|
m_activeFileName = inTesterOrOpt ? (m_fileName + "_optcache") : m_fileName;
|
|
m_activeFileCommon = !inTesterOrOpt;
|
|
//--- Claim these files before anything reads or writes them, and refuse to start if another chart in
|
|
//--- this terminal already holds them (see AcquireConfigLock). Deliberately placed here: this is the
|
|
//--- first moment the resolved filename - i.e. the config's true identity - is known, and it is still
|
|
//--- ahead of every load, seed and save. Tester/optimizer agents are exempt: each is a separate
|
|
//--- process writing its own sandboxed _optcache, and running one config across many agents in
|
|
//--- parallel is the entire point of an optimization.
|
|
#ifdef WARRIOR_EXPORT_FEATURES
|
|
//--- RESEARCH BUILD: no lock. This binary reads history and writes one CSV - it never trains, never saves
|
|
//--- a model (OnTick returns immediately, so no era ever completes) and therefore has nothing to protect
|
|
//--- against a concurrent chart. Claiming the lock would only make the exporter REFUSE to start whenever
|
|
//--- the config it wants to read is already open on a production chart, which is exactly when it is most
|
|
//--- useful to run.
|
|
#else
|
|
if(!inTesterOrOpt && !AcquireConfigLock())
|
|
return false;
|
|
#endif
|
|
//--- Any Strategy-Tester run - a single backtest OR an optimization pass - runs pure inference on the
|
|
//--- deployed model, never trains. A user optimizing TRADING parameters (SL/TP, filters, MM, ...) wants
|
|
//--- the AI held fixed at the deployed weights so passes are fast and comparable; retraining the net
|
|
//--- per config would be slow and make every pass a different model. AI hyperparameters are tuned on a
|
|
//--- chart (the internal auto-tuner / a real training run), not via MT5 optimization. Training and the
|
|
//--- new online continual-learning step therefore run ONLY on a live chart (see OnlineLearnStep).
|
|
m_inferenceOnly = MQLInfoInteger(MQL_TESTER);
|
|
//--- Seed the agent-local optcache from the deployed production model on the first tester/opt pass.
|
|
//--- Without this, the tester's separate _optcache file starts empty and the run retrains from zero -
|
|
//--- so a buyer who loads a .set and hits "backtest" waits through a full training run instead of a
|
|
//--- backtest of the model they deployed. The optcache shares the production model's exact config
|
|
//--- fingerprint (same m_fileName base), so the copied weights are guaranteed topology-compatible.
|
|
//--- Copies FROM FILE_COMMON (the live/manual-chart model) INTO the agent-local sandbox only; the
|
|
//--- production files are read, never written, so a backtest still can't corrupt the deployed model.
|
|
//--- Re-seeds when the cache is MISSING *or* STALE. Staleness matters because a tester run no longer
|
|
//--- writes this file at all (see PersistWeightsOnShutdown's inference-only skip), so without a
|
|
//--- freshness check the very first seeded copy would be reused forever - meaning the obvious workflow
|
|
//--- "retrain/redeploy on the chart, then backtest" would silently keep testing the OLD model. The
|
|
//--- config fingerprint in the filename can't catch this: retraining changes the WEIGHTS, not the
|
|
//--- topology inputs the fingerprint hashes, so the name stays identical.
|
|
bool cacheMissing = !FileIsExist(m_activeFileName + ".nnw");
|
|
bool cacheStale = false;
|
|
if(inTesterOrOpt && !cacheMissing && FileIsExist(m_fileName + ".nnw", FILE_COMMON))
|
|
{
|
|
datetime prodModified = (datetime)FileGetInteger(m_fileName + ".nnw", FILE_MODIFY_DATE, true);
|
|
datetime cacheModified = (datetime)FileGetInteger(m_activeFileName + ".nnw", FILE_MODIFY_DATE, false);
|
|
//--- both timestamps must be readable before trusting the comparison; a 0 means "couldn't tell",
|
|
//--- and re-seeding on an unreadable timestamp every single pass would be worse than not checking.
|
|
cacheStale = (prodModified > 0 && cacheModified > 0 && prodModified > cacheModified);
|
|
if(cacheStale)
|
|
Print(__FUNCTION__ + ": the deployed model is newer than this agent's cached copy - re-seeding so the backtest runs the CURRENT model, not the previously cached one.");
|
|
}
|
|
if(inTesterOrOpt && (cacheMissing || cacheStale))
|
|
{
|
|
if(FileIsExist(m_fileName + ".nnw", FILE_COMMON))
|
|
{
|
|
//--- The .nnw is the only copy that MUST succeed - retried (see CopyFileWithRetry's declaration
|
|
//--- comment) because a live chart's own atomic Save() can be mid-rename on this exact file.
|
|
//--- Its return value used to be ignored entirely, so a failed copy still logged "seeded tester
|
|
//--- cache..." as if it had worked, and the run silently trained from scratch instead.
|
|
if(CopyFileWithRetry(m_fileName + ".nnw", m_activeFileName + ".nnw"))
|
|
{
|
|
//--- Best-effort sidecars: not retried - losing one just means a cold calibration/shadow-blend
|
|
//--- start rather than a wrong/untrained model, which the .nnw copy above already guards against.
|
|
//--- Still share-aware (CopySharedFile, not FileCopy): the live chart holds these open too, so
|
|
//--- plain FileCopy would fail on them for exactly the same reason it failed on the .nnw.
|
|
if(FileIsExist(m_fileName + ".cfg", FILE_COMMON))
|
|
CopySharedFile(m_fileName + ".cfg", m_activeFileName + ".cfg", false);
|
|
if(FileIsExist(m_fileName + "_shadow.nnw", FILE_COMMON))
|
|
CopySharedFile(m_fileName + "_shadow.nnw", m_activeFileName + "_shadow.nnw", false);
|
|
//--- carry the calibration sidecar into the agent sandbox too, so a seeded backtest calibrates its
|
|
//--- live decisions with the deployed model's priors instead of the un-adjusted cold defaults.
|
|
if(FileIsExist(m_fileName + ".stats", FILE_COMMON))
|
|
CopySharedFile(m_fileName + ".stats", m_activeFileName + ".stats", false);
|
|
Print(__FUNCTION__ + ": seeded tester cache from the deployed production model (" + m_fileName + ") - this run reuses the deployed weights instead of retraining");
|
|
}
|
|
//--- else: CopyFileWithRetry already logged why. Fall through - the Net.Load() below will
|
|
//--- correctly report "no file" and BuildFreshTopology() takes over, same as a genuine first pass.
|
|
}
|
|
else if(m_inferenceOnly)
|
|
//--- Name the exact file (symbol + timeframe + config fingerprint) it looked for: the model is
|
|
//--- keyed on the CHART TIMEFRAME, so the #1 cause of this is running the tester on a different
|
|
//--- timeframe than the model was trained on (e.g. an H4 model, tester set to H1) - which reads
|
|
//--- as "no model" when one exists under a different timeframe. Spelling out the filename makes
|
|
//--- that mismatch obvious instead of looking like the deploy silently failed.
|
|
Print(__FUNCTION__ + ": WARNING - no deployed production model found at '" + m_fileName +
|
|
".nnw' (shared folder) for " + _Symbol + " " + EnumToString((ENUM_TIMEFRAMES)_Period) +
|
|
". A single backtest runs inference only and will NOT train. Most common cause: the tester" +
|
|
" timeframe differs from the one the model was trained on (the filename is keyed on timeframe)." +
|
|
" Otherwise, train this configuration on a chart first, then re-run the backtest.");
|
|
}
|
|
if(!LoadAndCompareTopologyConfiguration(m_activeFileName, m_initialNeuronsCount, m_hiddenLayersCount, m_neuronsReduction, m_minNeuronsCount, m_optimizationAlgo, m_historyBars, m_outputNeuronsCount, m_neuronsCount, m_minTrainYear, m_isInitialized, LEGACY_CONVERGE_WR_SLOT, m_fractalPeriods, m_convFilterCount, m_lstmHiddenSize, m_activeFileCommon))
|
|
{
|
|
// Topology/input params diverged from what produced the saved .nnw (or no .cfg exists yet;
|
|
// for inTesterOrOpt this is also the normal "first pass on this agent" case). The stale .cfg
|
|
// was already deleted on a mismatch, but the .nnw weights themselves are shaped for the OLD
|
|
// topology - loading them into a network built to the NEW shape would corrupt state or crash.
|
|
// Drop the incompatible weights/checkpoint too so the Net.Load() below cleanly misses and
|
|
// BuildFreshTopology() takes over (i.e. this pass pays the training cost once, and the result
|
|
// gets cached below for the NEXT pass to reuse, same as a live topology change would).
|
|
if(FileIsExist(m_activeFileName + ".nnw", m_activeFileCommon ? FILE_COMMON : 0))
|
|
{
|
|
Print(__FUNCTION__ + ": " + m_activeFileName + " - topology/input params changed since last save; discarding incompatible saved weights and starting fresh");
|
|
FileDelete(m_activeFileName + ".nnw", m_activeFileCommon ? FILE_COMMON : 0);
|
|
//--- Reaching here means a TRAINED model was just thrown away, so its drawn signals are stale for
|
|
//--- exactly the same reason ResetWeights() clears them: they would otherwise be restored moments
|
|
//--- later (LoadChartSignals runs at the end of this function) and shown as if they belonged to the
|
|
//--- model about to be trained. The sibling "no .cfg yet" case does not reach here (there are no
|
|
//--- weights to delete), so it is handled separately at the fresh-topology branch below.
|
|
ClearPersistedChartSignals("saved weights discarded - topology/input params changed");
|
|
}
|
|
if(FileIsExist(m_activeFileName + "_ckpt.tmp", m_activeFileCommon ? FILE_COMMON : 0))
|
|
FileDelete(m_activeFileName + "_ckpt.tmp", m_activeFileCommon ? FILE_COMMON : 0);
|
|
// Same reasoning applies to the EMA shadow-weight file (see m_shadowNet's declaration comment) -
|
|
// it's shaped for the OLD topology too, and EnsureShadowNet() has no independent way to detect
|
|
// that mismatch on Load() (CNet::Load() doesn't cross-validate against an expected shape). Drop
|
|
// it so EnsureShadowNet() cleanly misses and re-bootstraps from the fresh Net instead.
|
|
if(FileIsExist(m_activeFileName + "_shadow.nnw", m_activeFileCommon ? FILE_COMMON : 0))
|
|
FileDelete(m_activeFileName + "_shadow.nnw", m_activeFileCommon ? FILE_COMMON : 0);
|
|
//--- the calibration sidecar is tied to the discarded weights - drop it too so a fresh run
|
|
//--- re-measures priors from scratch instead of adjusting with a stale model's base rates.
|
|
if(FileIsExist(m_activeFileName + ".stats", m_activeFileCommon ? FILE_COMMON : 0))
|
|
FileDelete(m_activeFileName + ".stats", m_activeFileCommon ? FILE_COMMON : 0);
|
|
SaveTopologyConfiguration(m_activeFileName, m_initialNeuronsCount, m_hiddenLayersCount, m_neuronsReduction, m_minNeuronsCount, m_optimizationAlgo, m_historyBars, m_outputNeuronsCount, m_neuronsCount, LEGACY_STUDY_PERIOD_SLOT, m_minTrainYear, m_isInitialized, LEGACY_CONVERGE_WR_SLOT, m_fractalPeriods, m_convFilterCount, m_lstmHiddenSize, m_activeFileCommon);
|
|
}
|
|
double loadedIndicatorParams[];
|
|
//--- Inference-only backtest: if this deployed model was validated MQL5-inference-safe at deploy
|
|
//--- (marker in its .stats), load it host-only and run the pure-MQL5 forward path so the backtest
|
|
//--- never loads WarriorDML/WarriorCPU.dll - no DLL file-lock class of failure, and the exact math
|
|
//--- the Market build ships. Falls back to a compute backend just below if that load fails.
|
|
if(m_inferenceOnly && CheckPointer(Net) != POINTER_INVALID)
|
|
{
|
|
LoadModelStats(m_activeFileName, m_activeFileCommon); // reads m_mqlInferenceValidated (and priors)
|
|
if(m_mqlInferenceValidated)
|
|
{
|
|
Net.SetCpuInference(true);
|
|
PrintVerbose(__FUNCTION__ + ": " + ID + " - inference-only backtest running pure-MQL5 (DLL-free): the deployed model is validated MQL5-inference-safe");
|
|
}
|
|
}
|
|
bool netLoaded = LoadNetWithRetry(loadedIndicatorParams);
|
|
//--- Pure-MQL5 load failed unexpectedly (should not happen for a validated model) - drop back to a
|
|
//--- compute backend and retry once so the backtest still runs via the DLL rather than on a fresh net.
|
|
if(!netLoaded && CheckPointer(Net) != POINTER_INVALID && Net.CpuInference())
|
|
{
|
|
Print(__FUNCTION__ + ": " + ID + " - pure-MQL5 load failed; retrying with a compute backend (DLL)");
|
|
Net.SetCpuInference(false);
|
|
netLoaded = LoadNetWithRetry(loadedIndicatorParams);
|
|
}
|
|
//--- the file may carry a superseded architecture - correct it before anything reads the net
|
|
if(netLoaded)
|
|
EnforceTopologyContract();
|
|
//--- A superseded conv receptive field cannot be repaired in place (different weight-tensor shape), so
|
|
//--- the loaded net is discarded and the fresh-topology path below rebuilds and retrains. Deliberately
|
|
//--- routed through netLoaded rather than a separate branch: that path already cools the calibration,
|
|
//--- resets the era/trainingComplete state and re-arms the label-cache prebuild, all of which a genuine
|
|
//--- architecture change needs too.
|
|
if(netLoaded && m_topologySuperseded)
|
|
netLoaded = false;
|
|
//--- restore the calibration sidecar (priors + confidence scale) that pairs with these weights, so a
|
|
//--- restart - including a buyer's inference-only backtest - calibrates live decisions exactly as the
|
|
//--- saved model did instead of running with cold defaults (priors 0 => no adjustment). See LoadModelStats().
|
|
if(netLoaded)
|
|
LoadModelStats(m_activeFileName, m_activeFileCommon);
|
|
m_modelLoadedFromDisk = netLoaded;
|
|
//--- Make a successful resume visible (the counterpart to the fresh-start / mismatch messages below):
|
|
//--- on a live chart this confirms the saved model was found and loaded rather than silently retrained.
|
|
if(netLoaded && !inTesterOrOpt)
|
|
Print(ID + ": resumed saved model from era " + IntegerToString(m_eraCount) + " (trainingComplete=" + (string)m_trainingComplete + ") - continuing, not retraining from era 0.");
|
|
//--- a restart that recovers existing weights already has a proven-synced history and (since it
|
|
//--- already has at least one trained era behind it) doesn't have era 0's cold-start oversampling
|
|
//--- problem either - only a genuinely fresh start needs the 3 warm-up passes (see Train()'s
|
|
//--- m_warmupPassesRemaining gate).
|
|
//--- The label cache itself, however, is NEVER restored from the .nnw checkpoint - it lives only in
|
|
//--- the in-memory m_labelCacheBuy/Sell/HasValue arrays, which start empty every process start
|
|
//--- regardless of netLoaded. Previously this was set to netLoaded, which on a successful checkpoint
|
|
//--- load skipped the eager StartLabelCachePrebuild()/AdvanceLabelCachePrebuild() scan (Train()'s
|
|
//--- !m_labelCachePrebuilt gate) - every bar then fell through to the lazy per-bar fallback in the
|
|
//--- era loop, which calls ComputeLabelForBar(), a dead stub that unconditionally returns
|
|
//--- buy=false/sell=false (the real labeling logic lives ONLY in
|
|
//--- AdvanceBarrierLabelState(), reachable exclusively from the eager prebuild). The result: every
|
|
//--- restart that loaded a checkpoint silently force-labeled the entire era Neutral until something
|
|
//--- else (a topology mismatch, a fresh start) triggered a real prebuild. Always eager-prebuilding
|
|
//--- now, checkpoint or not, closes this at the root - the dead stub fallback then never matters.
|
|
m_warmupPassesRemaining = netLoaded ? 0 : 3;
|
|
m_labelCachePrebuilt = false;
|
|
if(inTesterOrOpt && netLoaded)
|
|
Print(__FUNCTION__ + ": " + ID + " - reused cached weights from a previous optimization/tester pass on this agent (era " + IntegerToString(m_eraCount) + ", trainingComplete=" + (string)m_trainingComplete + ") - skipping redundant training for this unchanged config");
|
|
if(netLoaded && ArraySize(loadedIndicatorParams) == AD_TUNE_PARAM_COUNT)
|
|
{
|
|
// Restart deploying previously AutoTune-d indicator params even with AutoTuneIndicators=false now -
|
|
// indicators above were already created with today's defaults, so rebuild them once with the
|
|
// restored values before any training/signal work happens.
|
|
m_indicatorTuner.Unflatten(loadedIndicatorParams);
|
|
ReInitADIndicators(indicators);
|
|
}
|
|
if(!netLoaded)
|
|
{
|
|
int error_code = GetLastError();
|
|
//--- Do NOT present error_code as the cause: on a no-GPU/CPU-DLL box it is the harmless 5100
|
|
//--- (OpenCL-not-found) left by the compute probe inside CNet::Load, NOT the reason the file was
|
|
//--- rejected. CNet::Load now prints the precise reason (bad marker / type mismatch / 0-layer stub /
|
|
//--- partial layer load) itself. Only clear the stale code here; the "rebuilding fresh" line below is
|
|
//--- the user-facing summary.
|
|
if(error_code != 5004) // not "file not found"
|
|
ResetLastError();
|
|
//--- CRITICAL: a failed load may have ALREADY overwritten the training-state out-params from the
|
|
//--- bad file's header before it was rejected - notably a corrupt/empty 0-layer stub whose header
|
|
//--- still says trainingComplete=1 (see CNet::Load's 0-layer guard). Left as-is, the freshly built,
|
|
//--- untrained topology below would be treated as an already-deployed converged model: it would
|
|
//--- never train, run inference on random weights (every bar scores Neutral, so the end-of-era NMS
|
|
//--- sweep deletes every chart arrow), and "save weights" would just re-persist that empty net.
|
|
//--- Force the state back to a genuine fresh start so BuildFreshTopology() actually gets trained.
|
|
m_trainingComplete = false;
|
|
m_eraCount = 0;
|
|
dtStudied = 0;
|
|
dForecast = 0;
|
|
//--- Cold the in-memory calibration so the freshly-rebuilt (untrained) topology below runs with no
|
|
//--- stale prior-correction until a retrain re-measures it (priors 0 => AdjustedSignalFromSoftmax is a
|
|
//--- no-op; scale 1.0 = the constructor default). In-memory ONLY - deliberately does NOT touch any
|
|
//--- file. (Earlier this session I also deleted the .stats/_shadow.nnw sidecars here; that was too
|
|
//--- destructive - a load failure can be transient/spurious (a not-yet-ready compute backend, a
|
|
//--- momentary file lock, or - on this CPU-DLL machine - GetLastError() being polluted with the
|
|
//--- harmless OpenCL-not-found 5100 from the probe inside CNet::Load), and wiping a user's calibration
|
|
//--- and deployed shadow on any such hiccup is the wrong default. The sidecars self-heal anyway: the
|
|
//--- shadow re-blends toward the retrained Net and .stats is overwritten on the next save.)
|
|
m_priorBuy = 0.0;
|
|
m_priorSell = 0.0;
|
|
m_priorNeutral = 0.0;
|
|
m_confidenceCalScale = 1.0;
|
|
//--- Accurate diagnostic (do NOT cite GetLastError() - inside CNet::Load the OpenCL probe leaves 5100
|
|
//--- there on a no-GPU/CPU-DLL box, which has nothing to do with the file). Distinguish an ordinary
|
|
//--- fresh start (no file yet) from a real read failure of an existing file by testing existence.
|
|
if(!inTesterOrOpt)
|
|
{
|
|
int loadFlags = m_activeFileCommon ? FILE_COMMON : 0;
|
|
if(FileIsExist(m_activeFileName + ".nnw", loadFlags))
|
|
Print(ID + ": could not read the existing model file " + m_activeFileName + ".nnw - rebuilding a fresh topology to retrain from era 0. Existing .stats/_shadow.nnw are KEPT (they refresh as training runs). If this recurs, that .nnw is likely corrupt - back it up, then use the panel's reset-weights to start clean.");
|
|
else
|
|
Print(ID + ": no saved model for this config yet - starting a fresh training run from era 0.");
|
|
}
|
|
//--- Re-seed before building a fresh topology so weight init is genuinely random. A prior
|
|
//--- genetic tuner sweep (TuneIndicatorsAndTrain's candidate eval loop, line ~5006) left the
|
|
//--- MQL5 RNG to a fixed seed; if this load-fail path then builds a production
|
|
//--- topology without re-seeding, the deployed model's weights would be deterministic/repeatable
|
|
//--- from whatever the last candidate's seed was — silently reproducible, not genuinely random.
|
|
//--- Matches ResetWeights() and Warrior_EA.mq5's OnInit.
|
|
MathSrand(GetTickCount());
|
|
//--- Era 0 with no weights behind it, so any arrow currently on this chart was drawn by a
|
|
//--- DIFFERENT model - the previous fingerprint's, or a corrupt .nnw's. Neither the panel reset
|
|
//--- nor the topology-mismatch discard above covers this path: both are gated on there being a
|
|
//--- saved .nnw to delete, and here there is none (a changed config produces a new m_fileName,
|
|
//--- so the old model's files are not "discarded", they are simply not this model's files).
|
|
//--- Left alone the stale arrows do NOT just look wrong - the chart objects survive deploys and
|
|
//--- restarts on their own, and SaveChartSignals() rebuilds the sidecar by scanning the chart,
|
|
//--- so the first save of this fresh run would adopt the dead model's calls as its own history.
|
|
//--- Deliberately at this call site rather than inside BuildFreshTopology(): the genetic tuner
|
|
//--- calls that for every throwaway candidate (AutoTune.mqh) and must not touch the chart.
|
|
ClearPersistedChartSignals("fresh topology at era 0 - arrows belong to a previous model");
|
|
if(!BuildFreshTopology())
|
|
return false;
|
|
}
|
|
TempData = new CArrayDouble();
|
|
if(CheckPointer(TempData) == POINTER_INVALID)
|
|
return false;
|
|
if(netLoaded)
|
|
// Populate dPrevSignal from the just-loaded weights immediately, rather than leaving it at
|
|
// its blank constructor default until the next (asynchronous, queued) training pass happens
|
|
// to run - matters most for the tester cache-reuse path above, where training may be skipped
|
|
// entirely for this run because dtStudied already covers the whole backtest window.
|
|
RefreshLatestSignal();
|
|
//--- Status line must match what the gate below (if(!m_trainingComplete && !m_inferenceOnly)) will
|
|
//--- actually do - otherwise an inference-only single backtest logs "resuming full training now" right
|
|
//--- under the "runs inference only and will NOT train" warning, which reads as a contradiction.
|
|
string trainState = m_trainingComplete
|
|
? "already complete - staying converged, no full retrain on this restart"
|
|
: (m_inferenceOnly
|
|
? "NOT complete, but this is an inference-only backtest - NOT training (see warning above); deploy a trained model for meaningful results"
|
|
: "NOT complete (interrupted or never converged) - resuming full training now");
|
|
Print(__FUNCTION__ + ": " + m_activeFileName + " - training " + trainState);
|
|
//--- Only kick off a full Train() run here if the loaded model genuinely isn't converged yet - an
|
|
//--- already-complete model used to get one full era-loop retrain (real Net.backProp() over the
|
|
//--- whole IS window) on every single EA restart/reattach for no reason, since this "Init" event
|
|
//--- bypassed ScheduleTrainingIfNeeded()'s m_trainingComplete gate entirely. dPrevSignal is already
|
|
//--- fresh from RefreshLatestSignal() above; ScheduleTrainingIfNeeded()'s normal per-tick check
|
|
//--- will call RefreshConvergedSignal() itself once a genuinely new bar closes.
|
|
if(!m_trainingComplete && !m_inferenceOnly)
|
|
bEventStudy = EventChartCustom(ChartID(), 1, (long)MathMax(0, MathMin(iTime(_Symbol, PERIOD_CURRENT, (int)(100 * Net.recentAverageSmoothingFactor * (m_trainingComplete ? 1 : 10))), dtStudied)), 0, "Init");
|
|
//--- Restore arrows persisted from a previous session (see SaveChartSignals). MUST run here, not in
|
|
//--- InitIndicators(): the arrows file is keyed on the FULL m_fileName including the per-config
|
|
//--- fingerprint, which is only appended above - see the note left at InitIndicators()'s old call site.
|
|
LoadChartSignals();
|
|
//--- bootstrap (or restore) the EMA shadow net now rather than waiting for the first
|
|
//--- RefreshLatestSignal()/era-blend call to lazily trigger it - see m_shadowNet's declaration
|
|
//--- comment.
|
|
EnsureShadowNet();
|
|
m_isInitialized = true;
|
|
#ifdef WARRIOR_EXPORT_FEATURES
|
|
//--- Research build only. Runs here because this is the first point at which the indicators, the buffers
|
|
//--- and the derived barrier horizon are all settled, and it needs no model, no labels and no training.
|
|
ExportFeatureMatrix();
|
|
#endif
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Shared training-set size estimate - see the declaration comment. |
|
|
//+------------------------------------------------------------------+
|
|
double CExpertSignalAIBase::EstimatedInSampleBars(void) const
|
|
{
|
|
int secs = PeriodSeconds(m_period);
|
|
if(secs <= 0)
|
|
secs = PeriodSeconds(PERIOD_H1);
|
|
double barsPerYear = (SECONDS_PER_YEAR / (double)secs) * MARKET_OPEN_FRACTION;
|
|
double oosKept = (100.0 - (double)m_oosSplitPct) / 100.0;
|
|
//--- MEASURED from the symbol's real history, matching Train()'s window exactly (earliest available
|
|
//--- bar, floored by MinTrainYear) now that training covers everything available rather than a
|
|
//--- configured number of years. There used to be a hard rule against reading Bars() here, and it was
|
|
//--- right for what it guarded: what is downloaded grows over a terminal's lifetime, and a topology
|
|
//--- that silently widens as history fills in would re-key its own weights file and throw away a
|
|
//--- trained model. That hazard is now closed at the other end instead - the derived shape is written
|
|
//--- into the .cfg on first build and ADOPTED, not re-derived, on every subsequent load, and none of
|
|
//--- the derived values feed the weights-filename fingerprint any more. So this is measured once per
|
|
//--- model, at the moment the model is created, and never consulted again for an existing one.
|
|
datetime firstAvailableBar = (datetime)SeriesInfoInteger(_Symbol, m_period, SERIES_FIRSTDATE);
|
|
MqlDateTime floorTime;
|
|
TimeCurrent(floorTime);
|
|
floorTime.year = m_minTrainYear;
|
|
floorTime.mon = 1;
|
|
floorTime.day = 1;
|
|
floorTime.hour = 0;
|
|
floorTime.min = 0;
|
|
floorTime.sec = 0;
|
|
datetime windowStart = StructToTime(floorTime);
|
|
if(firstAvailableBar > windowStart)
|
|
windowStart = firstAvailableBar;
|
|
int available = Bars(_Symbol, m_period, windowStart, TimeCurrent());
|
|
//--- History may not have finished syncing when a chart first attaches, and a model whose capacity was
|
|
//--- pinned from a handful of bars would stay crippled for its whole life - the one failure mode that
|
|
//--- measuring instead of assuming introduces. Fall back to a conservative fixed span rather than
|
|
//--- pinning something absurd, and say so, because the fix (reattach once history has synced) is the
|
|
//--- user's to make and is invisible otherwise.
|
|
if(available < TOPOLOGY_BUDGET_MIN_TRUSTED_BARS)
|
|
{
|
|
Print(ID + ": WARNING - only " + IntegerToString(available) + " bars of " + _Symbol +
|
|
" history are available yet, too few to size the network from. Falling back to a " +
|
|
IntegerToString(TOPOLOGY_BUDGET_FALLBACK_YEARS) + "-year assumption. If this is a fresh" +
|
|
" install, let the terminal finish downloading history and then delete this model's weights" +
|
|
" from the panel so the topology is sized from the real data.");
|
|
return (double)TOPOLOGY_BUDGET_FALLBACK_YEARS * barsPerYear * oosKept;
|
|
}
|
|
return (double)available * oosKept;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Dense-taper depth - see the declaration comment. |
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ComputeHiddenLayerCount(void) const
|
|
{
|
|
//--- Diagnostic escape hatch (compile-time, see ForceHiddenLayers). Deliberately not an input: this
|
|
//--- exists to run depth comparisons while working on the EA, and a user who picks a depth is
|
|
//--- contradicting the width and taper the code derived around it.
|
|
if(ForceHiddenLayers > 0)
|
|
return (int)MathMax(1, MathMin(MAX_HIDDEN_LAYERS, ForceHiddenLayers));
|
|
//--- Depth follows from the two ENDPOINTS the taper already has to connect - the derived first-layer
|
|
//--- width and the output-tied final hidden width (see BuildFreshTopology's taper block) - by asking
|
|
//--- how many steps it takes to get from one to the other at a sane per-layer compression ratio.
|
|
//--- Picking depth independently of those endpoints is what made it meaningless as an input: at 64
|
|
//--- units tapering to 12, four layers compress by 1.4x per step and five by barely 1.3x, so the extra
|
|
//--- depth buys no additional abstraction and costs a vanishing-gradient risk for nothing.
|
|
int lastHidden = (int)MathMax(HIDDEN_TAPER_OUTPUT_MULTIPLE * m_outputNeuronsCount, HIDDEN_TAPER_MIN_WIDTH);
|
|
lastHidden = (int)MathMin(lastHidden, m_initialNeuronsCount);
|
|
if(lastHidden <= 0 || m_initialNeuronsCount <= lastHidden)
|
|
return MIN_HIDDEN_LAYERS;
|
|
double steps = MathLog((double)m_initialNeuronsCount / (double)lastHidden) / MathLog(HIDDEN_TAPER_TARGET_RATIO);
|
|
int layers = (int)MathRound(steps) + 1; // +1: the first layer IS the starting endpoint, not a step
|
|
return (int)MathMax(MIN_HIDDEN_LAYERS, MathMin(MAX_HIDDEN_LAYERS, layers));
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Conv output-filter count - see the declaration comment. |
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ComputeConvFilterCount(void) const
|
|
{
|
|
//--- AddConvStage sets window = ConvReceptiveFieldBars() * m_neuronsCount and step = m_neuronsCount,
|
|
//--- so each sliding position covers that many BARS of features and the layer is a learned projection
|
|
//--- from the whole window down to this many filters. The meaningful reference point is therefore the
|
|
//--- WINDOW width, not one bar's feature count - budgeting against a single bar was correct only while
|
|
//--- the receptive field was 1, and at RF 3 it under-sized the stage 3x (8 filters for a 63-input
|
|
//--- window, an 8x squeeze, where the rule intends 2x). Same rule as before, applied to what the layer
|
|
//--- actually reads: halve the input. Keeping it tied to the window also means the receptive field and
|
|
//--- the filter count can never drift apart the way they did on 2026-07-31.
|
|
int chosen = (ConvReceptiveFieldBars() * m_neuronsCount) / CONV_COMPRESSION_DIVISOR;
|
|
//--- Snap DOWN to a power-of-two ladder for the same reason the first-layer width does: the target is
|
|
//--- approximate, and a value that moves with every feature toggle would re-key the weights file more
|
|
//--- often than the change in capacity justifies.
|
|
int ladder[] = {4, 8, 16, 32};
|
|
int snapped = CONV_FILTERS_MIN;
|
|
for(int i = 0; i < ArraySize(ladder); i++)
|
|
if(ladder[i] <= chosen)
|
|
snapped = ladder[i];
|
|
return (int)MathMax(CONV_FILTERS_MIN, MathMin(CONV_FILTERS_MAX, snapped));
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Derived front-end stages, for the startup config line. |
|
|
//+------------------------------------------------------------------+
|
|
string CExpertSignalAIBase::FrontEndConfigSummary(void) const
|
|
{
|
|
string s = "";
|
|
//--- conv slides a ConvReceptiveFieldBars()-bar window one bar at a time, emitting m_convFilterCount
|
|
//--- filters per position; the optional channel pool + second conv follow. Reported from the shape
|
|
//--- helpers rather than re-derived, so this line always describes what AddConvStage actually built.
|
|
if(UsesConvStage())
|
|
{
|
|
s += " | conv " + IntegerToString(ConvReceptiveFieldBars()) + " bars x" +
|
|
IntegerToString(m_neuronsCount) + "->" + IntegerToString(m_convFilterCount) +
|
|
" (" + IntegerToString(ConvFirstStagePositions()) + " pos)";
|
|
if(HasSecondConvStage())
|
|
s += " | pool /" + IntegerToString(m_convFilterCount) +
|
|
" | conv2 ->" + IntegerToString(ConvOutputPositions()) + " pos x" +
|
|
IntegerToString(m_convFilterCount) + " = " + IntegerToString(ConvOutputWidth());
|
|
else
|
|
s += " = " + IntegerToString(ConvOutputWidth());
|
|
}
|
|
if(UsesLstmStage())
|
|
s += " | lstm " + IntegerToString(LstmFanIn()) + "->" + IntegerToString(m_lstmHiddenSize);
|
|
//--- The dense stack is budgeted against the RAW input, so on any topology with a front-end it can be
|
|
//--- WIDER than the vector reaching it - a linear fan-out that cannot recover information the
|
|
//--- bottleneck already discarded, only add parameters. Flag it rather than silently reshaping a
|
|
//--- trained topology; see ComputeFirstLayerWidth.
|
|
int frontEndOut = UsesLstmStage() ? m_lstmHiddenSize
|
|
: (UsesConvStage() ? ConvOutputWidth() : 0);
|
|
if(frontEndOut > 0 && m_initialNeuronsCount > frontEndOut)
|
|
s += " | NOTE dense fans out " + IntegerToString(frontEndOut) + "->" +
|
|
IntegerToString(m_initialNeuronsCount);
|
|
return s;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Input width the LSTM block actually receives. |
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::LstmFanIn(void) const
|
|
{
|
|
//--- LSTM-only: the layer sits directly on the input, so it sees the whole flattened vector.
|
|
//--- HYBRID: AddConvStage runs first, so the LSTM sees the CONV FEATURE MAP, not the input. That map
|
|
//--- is position-major - ConvOutputPositions() positions of m_convFilterCount filters each - so the
|
|
//--- LSTM's per-timestep width stays m_convFilterCount (AddLstmStage) and its step count is the
|
|
//--- POSITION count, which the conv chain shrinks below m_historyBars once a multi-bar window and a
|
|
//--- second conv are in play. Hardcoding historyBars here would over-state the fan-in and, worse,
|
|
//--- disagree with the width CNet actually hands the layer.
|
|
//--- Budgeting HYBRID's LSTM against the flattened 420 UNDER-sized it by a full ladder step: the
|
|
//--- quadratic in ComputeLstmHiddenSize is dominated by the inputs term, so overstating the fan-in
|
|
//--- buys a smaller H for no reason.
|
|
//--- Requires m_convFilterCount to be settled first - InitNeuralNetwork orders it that way.
|
|
if(HasConvBeforeLstm())
|
|
return ConvOutputWidth();
|
|
return (int)m_historyBars * m_neuronsCount;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| LSTM recurrent hidden width - see the declaration comment. |
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ComputeLstmHiddenSize(void) const
|
|
{
|
|
//--- The LSTM block's parameter count is EXACTLY 4 * H * (H + inputs + 1) - see
|
|
//--- CNeuronLSTMOCL::SetInputs in AI\Network.mqh - and AddLstmStage feeds it the whole flattened
|
|
//--- input vector, so `inputs` is historyBars x neuronsCount. That makes this stage far and away the
|
|
//--- largest weight block in an LSTM or HYBRID model: at the shipped default of 32 units against a
|
|
//--- 540-wide input it is ~73k weights, more than DOUBLE the entire derived dense taper it feeds.
|
|
//--- It was the one part of the network the capacity budget never covered, which is why deriving the
|
|
//--- dense stack alone did not stop LSTM/HYBRID from being over-parameterized.
|
|
//--- Same budget as ComputeFirstLayerWidth: at most one weight per in-sample bar. Solving
|
|
//--- 4H(H+inputs+1) <= isBars for H is an ordinary quadratic, H = (-b + sqrt(b^2+4c))/2 with
|
|
//--- b = inputs+1 and c = isBars/4.
|
|
//--- PER-TIMESTEP width, not the flattened fan-in. The layer is now a recurrence: one shared gate
|
|
//--- block is applied at every step, so its parameter count is 4H(H + stepWidth + 1) - the whole
|
|
//--- point of weight sharing. Budgeting against the flattened width (420, or 160 behind conv) was
|
|
//--- correct for the old single-timestep layer and is now ~20x too pessimistic, which would starve
|
|
//--- the recurrence of hidden units for no reason.
|
|
//--- Must match what the layer is actually built as: a recurrence sizes its shared gate block on the
|
|
//--- PER-TIMESTEP width, while the single-timestep layer reads the whole flattened vector at once.
|
|
//--- Budgeting one against the other over-parameterizes by ~20x in one direction and starves the
|
|
//--- recurrence in the other. See LSTM_SEQUENCE_MODE.
|
|
int inputs = (LSTM_SEQUENCE_MODE ? (HasConvBeforeLstm() ? m_convFilterCount : m_neuronsCount)
|
|
: LstmFanIn());
|
|
double isBars = EstimatedInSampleBars();
|
|
if(inputs <= 0 || isBars <= 0.0)
|
|
return LSTM_HIDDEN_MIN;
|
|
double b = (double)(inputs + 1);
|
|
double budget = (-b + MathSqrt(b * b + 4.0 * (isBars / 4.0))) / 2.0;
|
|
int ladder[] = {8, 16, 32, 64, 128};
|
|
int snapped = LSTM_HIDDEN_MIN;
|
|
for(int i = 0; i < ArraySize(ladder); i++)
|
|
if((double)ladder[i] <= budget)
|
|
snapped = ladder[i];
|
|
return (int)MathMax(LSTM_HIDDEN_MIN, MathMin(LSTM_HIDDEN_MAX, snapped));
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Capacity budget for the first dense layer - see the declaration. |
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ComputeFirstLayerWidth(void) const
|
|
{
|
|
int inputWidth = (int)m_historyBars * m_neuronsCount;
|
|
if(inputWidth <= 0)
|
|
return FIRST_LAYER_MIN_WIDTH;
|
|
double isBars = EstimatedInSampleBars();
|
|
//--- One first-layer weight per in-sample bar. That layer is (inputWidth+1) x width and dominates the
|
|
//--- model, so this is effectively a whole-model capacity budget. One parameter per sample is already
|
|
//--- generous for a signal this weak; it is a ceiling, not a target.
|
|
int budget = (int)(isBars / (double)(inputWidth + 1));
|
|
//--- Snap DOWN to the ladder: the estimate above is approximate, and a value that moves with every
|
|
//--- small change would re-key the weights file for no benefit. Rungs are far enough apart that the
|
|
//--- estimate would have to be wrong by ~2x to land on a different one.
|
|
int ladder[] = {16, 32, 64, 128, 256, 512, 1024};
|
|
int chosen = FIRST_LAYER_MIN_WIDTH;
|
|
for(int i = 0; i < ArraySize(ladder); i++)
|
|
if(ladder[i] <= budget)
|
|
chosen = ladder[i];
|
|
//--- Budget below the floor means this configuration cannot support even the narrowest usable layer -
|
|
//--- the model will be over-parameterized no matter what is chosen here, and no amount of
|
|
//--- regularization fixes having more weights than examples. Typical cause is a high timeframe
|
|
//--- (D1 over 10 years is under 2,000 bars) or too many features for the history available. Say so:
|
|
//--- the fix is fewer HistoryBars / fewer feature groups / a longer study period, none of which this
|
|
//--- function can choose on the user's behalf.
|
|
if(budget < FIRST_LAYER_MIN_WIDTH)
|
|
Print(ID + ": WARNING - " + IntegerToString((int)isBars) + " estimated in-sample bars cannot support a " +
|
|
IntegerToString(inputWidth) + "-wide input. The first layer is being floored at " +
|
|
IntegerToString(FIRST_LAYER_MIN_WIDTH) + " units, which is still roughly " +
|
|
DoubleToString((double)(inputWidth + 1) * FIRST_LAYER_MIN_WIDTH / MathMax(1.0, isBars), 1) +
|
|
" weights per training bar - expect overfitting. Reduce HistoryBars or the feature set," +
|
|
" lengthen the study period, or train on a lower timeframe.");
|
|
return MathMax(FIRST_LAYER_MIN_WIDTH, chosen);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Batch-normalization layer - see the declaration comment. |
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::AddBatchNormStage(CArrayObj *topology, int units)
|
|
{
|
|
if(CheckPointer(topology) == POINTER_INVALID)
|
|
return false;
|
|
//--- Not an error: the input is off, so the topology simply has no normalization layers. Returning
|
|
//--- true keeps every call site a plain `if(!Add...) return false;` with no extra branching.
|
|
if(!EnableBatchNorm)
|
|
return true;
|
|
//--- A window of 1 makes the layer a no-op passthrough (mean==x, variance==0), which is a silently
|
|
//--- useless layer rather than an obviously absent one. Refuse to build it instead.
|
|
if(BatchNormWindow <= 1)
|
|
return true;
|
|
CLayerDescription *desc = new CLayerDescription();
|
|
if(CheckPointer(desc) == POINTER_INVALID)
|
|
return false;
|
|
desc.count = units;
|
|
desc.type = defNeuronBatchNorm;
|
|
desc.batch = BatchNormWindow;
|
|
//--- Identity forward transform. The non-linearity belongs to the dense layer stacked on top of this
|
|
//--- one; normalizing and then squashing in the same step would undo the normalization.
|
|
desc.activation = NONE;
|
|
desc.optimization = (ENUM_OPTIMIZATION)m_optimizationAlgo;
|
|
if(!topology.Add(desc))
|
|
{
|
|
delete desc;
|
|
return false;
|
|
}
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Convolution front-end: conv -> channel pool -> conv. Shared by |
|
|
//| CSignalCONV and CSignalHYBRID - see the declaration comment. |
|
|
//| |
|
|
//| MEMORY LAYOUT, which is what every decision here turns on: |
|
|
//| - The INPUT is bar-major: BufferTempData appends m_neuronsCount |
|
|
//| contiguous features per bar, bars in order. So a flat window of |
|
|
//| k*m_neuronsCount spans exactly k consecutive BARS, and a step |
|
|
//| of m_neuronsCount advances exactly one bar. A multi-bar |
|
|
//| receptive field therefore needs NO kernel change. |
|
|
//| - A CONV OUTPUT is position-major: FeedForwardConv (AI\Network.cl)|
|
|
//| emits matrix_o[out + window_out * i], so one position's |
|
|
//| window_out filter responses are CONTIGUOUS and consecutive |
|
|
//| positions sit window_out apart. |
|
|
//| - Both pool implementations (FeedForwardProof, CPU_FeedForwardProof)
|
|
//| slide FLAT: pos = i*step over `window` CONSECUTIVE elements. |
|
|
//| Over a position-major buffer those neighbours are the FILTERS of |
|
|
//| one position. So a pool here is a max-over-CHANNELS, never a |
|
|
//| pool across time. |
|
|
//| |
|
|
//| That is exactly the NeuroNet_DNG reference contract (see |
|
|
//| references\MQL5\Experts\EDL\Trajectory.mqh layers 2-5, kernels |
|
|
//| byte-identical to ours): conv(window=2, step=1, window_out=4) -> |
|
|
//| pool(window=4, step=4) -> conv -> pool. The pool is tied to the |
|
|
//| filter count, giving a clean non-overlapping channel reduction. |
|
|
//| |
|
|
//| WHY THE POOL WINDOW MUST STAY TIED TO window_out. Shipping |
|
|
//| window=3/step=2 against 16 filters overlapped windows across the |
|
|
//| filter axis and straddled position boundaries, collapsing |
|
|
//| unrelated detectors into whichever fired hardest below every |
|
|
//| learnable layer (CONV sat at ~40% balanced accuracy for 510 eras, |
|
|
//| Sell recall 0%). The opposite error is just as bad: dropping the |
|
|
//| pool but leaving conv at window=step=one bar is a 1x1 conv that |
|
|
//| never mixes across time at all. |
|
|
//| |
|
|
//| We stop one layer short of the reference and do NOT append the |
|
|
//| second pool. A channel pool emits one scalar per position, so a |
|
|
//| trailing pool would hand the dense stack ~18 values for a 420-wide |
|
|
//| input and force it to FAN OUT 18 -> 64 instead of funnelling. The |
|
|
//| reference affords that at window_out=4 against a far smaller |
|
|
//| input; here it is a bottleneck below every learnable layer. The |
|
|
//| full 8-filter map goes to the dense/LSTM stage. |
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::AddConvStage(CArrayObj *topology)
|
|
{
|
|
if(CheckPointer(topology) == POINTER_INVALID)
|
|
return false;
|
|
//--- Stage 1: convolution across CONV_RECEPTIVE_FIELD_BARS bars, advancing one bar at a time.
|
|
CLayerDescription *desc = new CLayerDescription();
|
|
if(CheckPointer(desc) == POINTER_INVALID)
|
|
return false;
|
|
//--- desc.count here is the conv layer's own output-filter count (CNeuronConvOCL::Init's window_out
|
|
//--- param, AI\Network.mqh) - was m_hiddenLayersCount (an unrelated dense-taper-depth setting,
|
|
//--- defaulting to 4), bottlenecking every sliding position to just 4 filters regardless of how wide
|
|
//--- the rest of the network was. See ConvFilterCount's declaration comment (Variables\Inputs.mqh).
|
|
desc.count = m_convFilterCount;
|
|
desc.type = defNeuronConv;
|
|
// PRELU, not TANH: matches what CNeuronConv's CPU path (Network.mqh) has always hardcoded
|
|
// regardless of this setting (its activationFunction() override ignores `activation` entirely) -
|
|
// this used to silently diverge from the GPU/DirectML tier, which DOES honor this field and was
|
|
// therefore actually running tanh instead of the intended PReLU whenever hardware accel was active.
|
|
desc.activation = PRELU;
|
|
desc.optimization = (ENUM_OPTIMIZATION)m_optimizationAlgo;
|
|
//--- The whole point: a window spanning several bars. Guarded because m_historyBars can be small
|
|
//--- enough that a multi-bar window would not fit at all, in which case this degrades to the old
|
|
//--- per-bar projection rather than building a negative-width layer.
|
|
desc.window = ConvReceptiveFieldBars() * m_neuronsCount;
|
|
desc.step = m_neuronsCount;
|
|
if(!topology.Add(desc))
|
|
{
|
|
delete desc;
|
|
return false;
|
|
}
|
|
//--- NO POOL, and no second conv. See CONV_RECEPTIVE_FIELD_BARS' comment for the measurement and the
|
|
//--- reference-kernel reading behind that: the conv emits position-major output and the reference pool is
|
|
//--- a flat contiguous max, so a pool here can only ever reduce ACROSS FILTERS within a position, never
|
|
//--- over time. It threw away 87.5% of this layer's output and starved every non-argmax filter of
|
|
//--- gradient. The second conv was mis-shaped in the same change - its window was counted in raw elements
|
|
//--- while its comment claimed positions, so a "2-position" window actually spanned 2 FILTERS of position
|
|
//--- 0 - and it only existed to consume the pool's output.
|
|
//--- If a deeper hierarchy is wanted later, the correct shape on THIS layout is a strided conv over
|
|
//--- positions: window = k * m_convFilterCount, step = s * m_convFilterCount (both whole numbers of
|
|
//--- positions, which IS contiguous in position-major order), never a pool. Springenberg et al. ICLR 2015.
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Conv chain shape. SINGLE SOURCE OF TRUTH - AddConvStage builds |
|
|
//| from these and LstmFanIn/FrontEndConfigSummary report from them, |
|
|
//| so what is constructed and what is logged cannot drift apart. |
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ConvReceptiveFieldBars(void) const
|
|
{
|
|
//--- Degrade to a per-bar projection rather than build an impossible layer when history is too short
|
|
//--- for a multi-bar window. MathMin against m_historyBars keeps window <= input width.
|
|
int bars = (int)MathMin((int)CONV_RECEPTIVE_FIELD_BARS, (int)m_historyBars);
|
|
return (bars > 0 ? bars : 1);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ConvFirstStagePositions(void) const
|
|
{
|
|
//--- Sliding positions of stage 1: window ConvReceptiveFieldBars() bars, step 1 bar.
|
|
int p = (int)m_historyBars - (ConvReceptiveFieldBars() - 1);
|
|
return (p > 0 ? p : 1);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::HasSecondConvStage(void) const
|
|
{
|
|
//--- Permanently false: the conv chain is ONE true convolution. Kept (rather than deleted along with the
|
|
//--- pool + second conv it used to gate) so ConvOutputPositions/ConvOutputWidth stay the single source of
|
|
//--- truth for the chain's shape and a future strided second stage has one place to switch itself on.
|
|
return false;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ConvOutputPositions(void) const
|
|
{
|
|
int p = ConvFirstStagePositions();
|
|
return (HasSecondConvStage() ? p - (ConvReceptiveFieldBars() - 1) : p);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ConvOutputWidth(void) const
|
|
{
|
|
//--- Total element count reaching whatever is stacked above the conv chain: the conv output is
|
|
//--- position-major, window_out filters per position.
|
|
return ConvOutputPositions() * m_convFilterCount;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| LSTM sequence stage. Shared by CSignalLSTM and CSignalHYBRID - |
|
|
//| see the declaration comment. |
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::AddLstmStage(CArrayObj *topology)
|
|
{
|
|
if(CheckPointer(topology) == POINTER_INVALID)
|
|
return false;
|
|
CLayerDescription *desc = new CLayerDescription();
|
|
if(CheckPointer(desc) == POINTER_INVALID)
|
|
return false;
|
|
desc.count = m_lstmHiddenSize;
|
|
desc.type = defNeuronLSTM;
|
|
desc.activation = TANH;
|
|
//--- CNeuronLSTMOCL now has an accelerated SGD+momentum kernel (LSTM_UpdateWeightsMomentum,
|
|
//--- AI\Network.mqh/Network.cl/DirectML\WarriorCPU.cpp/WarriorDML.cpp) alongside the original
|
|
//--- Adam one, so this layer honors the same TrainingOptimizer input as PAI/CONV - see
|
|
//--- m_optimizationAlgo's declaration comment.
|
|
desc.optimization = (ENUM_OPTIMIZATION)m_optimizationAlgo;
|
|
//--- PER-TIMESTEP input width - the feature count for ONE bar as it reaches this layer. CNet passes
|
|
//--- this to CNeuronLSTMOCL::SetStepWidth(), which is what makes the layer an actual recurrence over
|
|
//--- m_historyBars steps instead of a single gated projection over the whole flattened vector. It
|
|
//--- must divide LstmFanIn() exactly, which it does by construction in both placements: on the raw
|
|
//--- input the vector is historyBars x m_neuronsCount, and behind the conv chain it is
|
|
//--- ConvOutputPositions() x m_convFilterCount (position-major, filters contiguous per position -
|
|
//--- see the layout note above AddConvStage). Note the step COUNT is the position count, which the
|
|
//--- conv chain shrinks below historyBars once a multi-bar window and a second conv are in play.
|
|
//--- 0 disables sequence mode in CNeuronLSTMOCL::SetStepWidth (which maps <=0 to "not a sequence"),
|
|
//--- restoring the single-timestep layer. See LSTM_SEQUENCE_MODE.
|
|
desc.window = (LSTM_SEQUENCE_MODE ? (HasConvBeforeLstm() ? m_convFilterCount : m_neuronsCount) : 0);
|
|
//--- MathMax(1,...) guard taken from the HYBRID copy: the CSignalLSTM copy divided unguarded, so a
|
|
//--- historyBars of 1 produced step 0 there and step 1 here for what is meant to be the same layer.
|
|
desc.step = MathMax(1, (int)m_historyBars / 2);
|
|
if(!topology.Add(desc))
|
|
{
|
|
delete desc;
|
|
return false;
|
|
}
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Builds a fresh, untrained topology into Net - the exact layer |
|
|
//| construction InitNeuralNetwork() used to inline for the |
|
|
//| "no saved .nnw" case; factored out so TuneIndicatorsAndTrain() can|
|
|
//| get a clean-slate Net per trial without touching indicator init. |
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::BuildFreshTopology()
|
|
{
|
|
CArrayObj *Topology = new CArrayObj();
|
|
if(CheckPointer(Topology) == POINTER_INVALID)
|
|
return false;
|
|
//--- Input Layer
|
|
CLayerDescription *desc = new CLayerDescription();
|
|
if(CheckPointer(desc) == POINTER_INVALID)
|
|
{
|
|
delete Topology;
|
|
return false;
|
|
}
|
|
desc.count = m_historyBars * m_neuronsCount;
|
|
desc.type = defNeuron;
|
|
desc.activation = NONE;
|
|
desc.optimization = (ENUM_OPTIMIZATION)m_optimizationAlgo;
|
|
if(!Topology.Add(desc))
|
|
{
|
|
delete Topology;
|
|
return false;
|
|
}
|
|
//--- neuron-type-specific layers (Conv+Pool, LSTM, or none for a plain perceptron)
|
|
if(!AddCustomLayers(Topology))
|
|
{
|
|
delete Topology;
|
|
return false;
|
|
}
|
|
//--- Hidden Layers, tapering from m_initialNeuronsCount down to m_minNeuronsCount, each preceded by
|
|
//--- a batch-normalization layer (no-op when EnableBatchNorm is off). Placed BETWEEN layers rather
|
|
//--- than inside them because every layer here computes activation(W.x+b) in a single kernel - there
|
|
//--- is no seam between the matmul and the non-linearity to insert anything into. Normalizing the
|
|
//--- previous layer's OUTPUT is the equivalent formulation and is exactly what the NeuroNet_DNG
|
|
//--- reference's own worked example does (input -> BatchNorm -> hidden -> output).
|
|
//--- The first one also normalizes whatever the conv/pool/LSTM stage produced, which is the widest
|
|
//--- unbounded stage in the whole network and the one whose scale drift hurts most.
|
|
//--- GEOMETRIC taper from the derived first-layer width down to a final hidden width tied to the
|
|
//--- output count, spread evenly over however many layers the architecture asks for. This replaces a
|
|
//--- pair of inputs (NeuronsReduction, MinNeuronsCount) that were calibrated when the first layer was
|
|
//--- a hand-picked 500: they produced a genuine 500 -> 150 -> 45 funnel there, but against the derived
|
|
//--- width they degenerate. At 64 units, "keep 30% with a floor of 20" gives 64 -> 20 -> 20 - the
|
|
//--- reduction stops mattering after one step and the "minimum" silently becomes the width of every
|
|
//--- layer but the first. Deriving the ratio from the endpoints keeps the funnel shape correct at any
|
|
//--- width, which is the whole point of having derived the width in the first place.
|
|
int lastHidden = MathMax(HIDDEN_TAPER_OUTPUT_MULTIPLE * m_outputNeuronsCount, HIDDEN_TAPER_MIN_WIDTH);
|
|
//--- Never wider than where the taper starts: a narrow first layer (see the D1 case in
|
|
//--- ComputeFirstLayerWidth) must still funnel DOWN, not fan back out.
|
|
lastHidden = MathMin(lastHidden, m_initialNeuronsCount);
|
|
double taperRatio = (m_hiddenLayersCount > 1)
|
|
? MathPow((double)lastHidden / (double)m_initialNeuronsCount, 1.0 / (double)(m_hiddenLayersCount - 1))
|
|
: 1.0;
|
|
//--- Width of the layer immediately below the next batch-norm layer. Only advisory (CNet sizes each
|
|
//--- batch-norm layer from whatever it actually sits on), but kept honest so the descriptor list
|
|
//--- reads correctly. Seeded with the input width - which is also a lie for CONV/LSTM/HYBRID, where
|
|
//--- the custom stage in between has resized things; that is exactly why CNet does not trust it.
|
|
int prevWidth = (int)(m_historyBars * m_neuronsCount);
|
|
bool result = true;
|
|
for(int i = 0; (i < m_hiddenLayersCount && result); i++)
|
|
{
|
|
int n = (i == 0)
|
|
? m_initialNeuronsCount
|
|
: MathMax(lastHidden, (int)MathRound(m_initialNeuronsCount * MathPow(taperRatio, (double)i)));
|
|
result = (AddBatchNormStage(Topology, prevWidth) && result);
|
|
if(!result)
|
|
break;
|
|
prevWidth = n;
|
|
desc = new CLayerDescription();
|
|
if(CheckPointer(desc) == POINTER_INVALID)
|
|
{
|
|
delete Topology;
|
|
return false;
|
|
}
|
|
desc.count = n;
|
|
desc.type = defNeuron;
|
|
desc.activation = HiddenLayerActivation();
|
|
desc.optimization = (ENUM_OPTIMIZATION)m_optimizationAlgo;
|
|
result = (Topology.Add(desc) && result);
|
|
}
|
|
if(!result)
|
|
{
|
|
delete Topology;
|
|
return false;
|
|
}
|
|
//--- Batch norm immediately before the head. This is the one placement that matters most: it is what
|
|
//--- keeps the logit spread from decaying as the weights below it shrink, and it is the precondition
|
|
//--- for ever running an UNBOUNDED head here (see the 2026-07-28 note on desc.activation below).
|
|
if(!AddBatchNormStage(Topology, prevWidth))
|
|
{
|
|
delete Topology;
|
|
return false;
|
|
}
|
|
//--- Output Layer
|
|
desc = new CLayerDescription();
|
|
if(CheckPointer(desc) == POINTER_INVALID)
|
|
{
|
|
delete Topology;
|
|
return false;
|
|
}
|
|
desc.count = m_outputNeuronsCount;
|
|
desc.type = defNeuron;
|
|
// Never write the activation as a literal here: this line only ever reaches a BRAND-NEW topology, so
|
|
// a change made here never touches an existing .nnw (CNeuronBaseOCL::Save persists the activation and
|
|
// Load restores it). OutputLayerActivation() is the single source of truth and EnforceTopologyContract()
|
|
// re-asserts it after every Load.
|
|
// Regression (1 output): TANH - its [-1,1] range maps straight onto the -1/0/1 Sell/Neutral/Buy
|
|
// convention, with no SIGMOID offset or clipped ReLU half.
|
|
// Classification (3 outputs): SIGMOID, deliberately NOT NONE.
|
|
// The forward head must stay BOUNDED. HiddenLayerActivation() is PRELU, so this is the only bounded
|
|
// stage in the forward path, and two downstream constants are calibrated against that: CLASS_LOGIT_SCALE
|
|
// = 6.0 is a temperature gain sized to stretch [0,1] into a usable logit span (against a free logit it
|
|
// is just a 6x amplifier), and the +-3.0 cold-start bias seed reads "sigmoid(+-3) ~= 0.95/0.05". An
|
|
// unbounded head was tried 2026-07-27 and reverted the next day: it starts at exp(6*3) vs exp(6*-3),
|
|
// saturated softmax makes the gradient input-INDEPENDENT, and all four architectures degenerated to
|
|
// one- or two-class output within two eras. Do NOT unbound the head again without simultaneously
|
|
// setting CLASS_LOGIT_SCALE to 1.0 and the bias magnitude to ~0.5.
|
|
// The BACKWARD pass is not 3 independent sigmoid deltas: CNet::backProp/backPropOCL detect the
|
|
// 3-output case and compute a joint softmax + categorical-cross-entropy gradient (softmax_i - target_i),
|
|
// which is what ties the classes together - raising one probability structurally lowers the other two.
|
|
// ApplyClassificationSoftmax() reproduces exactly that normalization at read time.
|
|
desc.activation = OutputLayerActivation();
|
|
desc.optimization = (ENUM_OPTIMIZATION)m_optimizationAlgo;
|
|
if(!Topology.Add(desc))
|
|
{
|
|
delete Topology;
|
|
return false;
|
|
}
|
|
if(CheckPointer(Net) != POINTER_INVALID)
|
|
delete Net;
|
|
Net = new CNet(Topology);
|
|
delete Topology;
|
|
if(CheckPointer(Net) == POINTER_INVALID)
|
|
return false;
|
|
// A fresh topology invalidates any existing shadow (see m_shadowNet's declaration comment) -
|
|
// its weights, if any, are shaped for the OLD Net and would either mismatch dimensionally or,
|
|
// worse, silently blend unrelated weight spaces if the shape happens to coincide. Reset to NULL
|
|
// here; EnsureShadowNet() lazily re-bootstraps a fresh clone of the new Net on first use.
|
|
if(CheckPointer(m_shadowNet) != POINTER_INVALID)
|
|
{
|
|
delete m_shadowNet;
|
|
m_shadowNet = NULL;
|
|
}
|
|
//--- Let EnsureShadowNet() re-attempt the clone bootstrap once for this new topology (see the latch's
|
|
//--- declaration comment) - the old shadow, and any prior failed-bootstrap verdict, no longer apply.
|
|
m_shadowBootstrapAttempted = false;
|
|
//--- A brand-new untrained net has NO online continual-learning history (see OnlineLearnStep): reset
|
|
//--- the watermark/guardrail/counters so a fresh start or a ResetWeights()-then-retrain never resumes
|
|
//--- from a superseded model's learned-up-to point or its stale rolling accuracy. Restored (not reset)
|
|
//--- on a normal reload of an existing model - that path loads them via LoadModelStats() and never
|
|
//--- calls BuildFreshTopology(). Harmless during the post-tune rebuild (online learning is
|
|
//--- gated off there, and the deployed final retrain rebuilds and resets again before deployment).
|
|
m_onlineLearnedUpToTime = 0;
|
|
m_onlineRollingAcc = -1.0;
|
|
m_onlineSamples = 0;
|
|
m_onlineBarsSincePersist = 0;
|
|
m_onlineBlendFrozen = false;
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| |
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::InitIndicators(CIndicators *indicators)
|
|
{
|
|
//--- Reset only the status label on (re-)init; deliberately do NOT PurgeChart() here so previously drawn
|
|
//--- signal arrows survive an EA re-init (recompile / param change / timeframe switch) instead of
|
|
//--- vanishing every time - see SIG_ARROW_PREFIX. Full cleanup still happens in the destructor.
|
|
ClearStatusLabel();
|
|
//--- NOTE: LoadChartSignals() is deliberately NOT called here any more. This method runs from
|
|
//--- InitNeuralNetwork() BEFORE the per-config fingerprint is appended to m_fileName, so at this point
|
|
//--- m_fileName is only "<folder>\<symbol>_<period>" - the load looked for e.g. "SP500_16385.arrows"
|
|
//--- while SaveChartSignals() (which only ever runs post-init, with the finished name) had written
|
|
//--- "SP500_16385_3.00000000_1.00000000_<hash>.arrows". The mismatch made the restore silently no-op on
|
|
//--- every restart from the moment the fingerprint was introduced. It is now called at the END of
|
|
//--- InitNeuralNetwork(), once m_fileName is final. Same family as the fingerprint trap documented at
|
|
//--- BuildConfigFingerprint: anything keyed on m_fileName must run AFTER it is fully built.
|
|
if(!InitOpen(indicators))
|
|
return false;
|
|
if(!InitClose(indicators))
|
|
return false;
|
|
if(!InitLow(indicators))
|
|
return false;
|
|
if(!InitHigh(indicators))
|
|
return false;
|
|
//--- label source, always created unconditionally, same as the OHLC indicators above - see
|
|
//--- m_ADZigZag's declaration comment. Optionally ALSO read as an input feature (m_useSwingContext,
|
|
//--- below) using the same already-running indicator instance - no separate init needed for that.
|
|
if(!InitADZigZag(indicators))
|
|
return false;
|
|
m_neuronsCount = 4; // (close-open)/atr, (high-open)/atr, (low-open)/atr, bullish/bearish flag
|
|
if(m_useVolumes)
|
|
{
|
|
// change ratio, level vs 50-bar baseline, absorption (range per unit volume), volume x range -
|
|
// see BufferTempDataCompute()'s matching block, and research/test_volume.py for the measurement
|
|
// that justified widening this from 1. m_neuronsCount is already in the config fingerprint, so
|
|
// this re-keys existing caches on its own: correct, the input vector genuinely changed shape.
|
|
m_neuronsCount += 4;
|
|
if(!InitVolumes(indicators))
|
|
return false;
|
|
}
|
|
// Unconditional, same reasoning as m_ATR/m_ADZigZag below: m_Time.GetData() is read
|
|
// unconditionally elsewhere (label-eligibility gate, cache anchor, online-learning watermark,
|
|
// arrow timestamps) regardless of whether the cyclical time-of-day/day-of-week values are also
|
|
// opted into as an explicit feature via m_useTime - so the indicator itself must always exist.
|
|
if(!InitTime(indicators))
|
|
return false;
|
|
if(m_useTime)
|
|
{
|
|
m_neuronsCount += 6;
|
|
}
|
|
if(m_useATR)
|
|
{
|
|
//already init in the base class
|
|
m_neuronsCount++;
|
|
}
|
|
if(m_useMA)
|
|
{
|
|
if(!InitMA(indicators))
|
|
return false;
|
|
m_neuronsCount += 5; // (open-MA)/atr, (high-MA)/atr, (low-MA)/atr, (close-MA)/atr, (MA-MA[1])/atr
|
|
}
|
|
if(m_useRSI)
|
|
{
|
|
if(!InitRSI(indicators))
|
|
return false;
|
|
m_neuronsCount++; // RSI/100
|
|
}
|
|
if(m_useMACD)
|
|
{
|
|
if(!InitMACDFeature(indicators))
|
|
return false;
|
|
m_neuronsCount += 3; // main/atr, signal/atr, histogram/atr
|
|
}
|
|
if(m_useIchimoku)
|
|
{
|
|
if(!InitIchimoku(indicators))
|
|
return false;
|
|
// (close-Tenkan)/atr, (close-Kijun)/atr, (Tenkan-Kijun)/atr, (close-SpanA)/atr, (close-SpanB)/atr,
|
|
// signed cloud thickness at this bar, signed PROJECTED cloud thickness, Chikou displacement
|
|
m_neuronsCount += 8;
|
|
}
|
|
if(m_useSwingContext)
|
|
m_neuronsCount += 9; // 5 confirmed-pivot features (direction, distance-since-pivot, prior-leg magnitude, retracement ratio, bars-since-pivot) + 4 recent-context features (Donchian pos 20/50, 20-bar return, 20-bar SMA extension) - see BufferTempDataCompute()'s matching block
|
|
if(m_useNews)
|
|
m_neuronsCount += 2; // NewsRecency, NewsProximity - see BufferTempDataCompute()'s matching block
|
|
if(m_useSpreadFeature)
|
|
m_neuronsCount += 2; // spread/ATR (volatility-regime reading), spread change ratio
|
|
// - see BufferTempDataCompute()'s matching block
|
|
if(m_useCrossAsset)
|
|
m_neuronsCount += CROSSASSET_FEATURES; // base/quote currency strength (fast+slow), pair-vs-currencies
|
|
// divergence, cross-sectional dispersion - System\CrossAsset.mqh
|
|
if(m_useADCumulativeDelta)
|
|
{
|
|
if(!InitADCumulativeDelta(indicators))
|
|
return false;
|
|
m_neuronsCount += 6; // Pressure, CumulativeDelta, BullishPressure, BearishPressure, Absorption, Initiative
|
|
}
|
|
if(m_useADShorteningOfThrust)
|
|
{
|
|
if(!InitADShorteningOfThrust(indicators))
|
|
return false;
|
|
m_neuronsCount += 4; // SOT, SOTEffortRegime, SOTConfirmation, SOTPushRegime
|
|
}
|
|
if(m_useADWyckoffEventStream)
|
|
{
|
|
if(!InitADWyckoffEventStream(indicators))
|
|
return false;
|
|
// 13, not 14 - only EventPrice (buffer 4) is excluded, see BufferTempData()'s comment (EventPhase,
|
|
// buffer 1, joined the feature set on 2026-08-02 when the indicator stopped writing it as a copy
|
|
// of EventCode)
|
|
m_neuronsCount += 13; // EventCode, EventPhase, ZoneTop, ZoneBottom, StructuralPhase, CHoCHTrendToRange, CHoCHRangeToTrend, SlopeAccumulationBullish, SlopeAccumulationBearish, SlopeDistributionBullish, SlopeDistributionBearish, Reaccumulation, Redistribution
|
|
}
|
|
if(m_useADWyckoffFailedStructure)
|
|
{
|
|
if(!InitADWyckoffFailedStructure(indicators))
|
|
return false;
|
|
m_neuronsCount += 5; // Value, BullishStructuralFailure, BearishStructuralFailure, FailedAccumulation, FailedDistribution
|
|
}
|
|
if(m_useADWyckoffSignificantBarInversion)
|
|
{
|
|
if(!InitADWyckoffSignificantBarInversion(indicators))
|
|
return false;
|
|
m_neuronsCount += 5; // SignificantBarQuality, BullishSignificantBar, BearishSignificantBar, BullishControlFlip, BearishControlFlip
|
|
}
|
|
if(!FolderCreate(m_folderPath, FILE_COMMON))
|
|
{
|
|
if(GetLastError() != 5010) // If the error is not because the folder already exists
|
|
{
|
|
Print("Failed to create folder: " + m_folderPath);
|
|
}
|
|
else
|
|
{
|
|
ResetLastError(); // Reset the error code
|
|
}
|
|
}
|
|
return true;
|
|
}
|
|
#endif
|