forked from animatedread/Warrior_EA
0adaea48b6
A resumed META model hot-looped pass 1 (0->100% scan oscillation, silent for
3 minutes until the stall reporter fired) because EVERY window failed at the
first AD/Wyckoff feature: the init-time param adoption called
ReInitADIndicators unconditionally, destroying five freshly-calculating
indicator instances to recreate them with BYTE-IDENTICAL params (verified by
parsing the .nnw header - the MI tuner had kept the configured settings), at
process start, on a box with 1 GB free of 31. The replacements sat cold for
6+ minutes while full-history resweeps starved the indicator threads harder.
- AdoptIndicatorParams: installs a loaded param set into the tuner and
rebuilds handles ONLY when the set actually differs from what the live
indicators run. Both call sites (resume init + panel reload) use it.
- Resumed models get the same 3 warm-up passes as fresh ones. The skip was
the shared root cause of the cold-ATR (ba13eef), cold-AD (2026-08-11) and
this incident - custom indicators recompute from scratch every process
start regardless of what the .nnw proves.
- Cold-sweep backoff: a pass-1 sweep in which every window failed on a
TRANSIENT cause arms a 5s era-start pause instead of an immediate
full-history resweep, so the retry loop stops consuming the CPU/memory the
warming indicators need. The stall reporter names the backoff branch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1516 lines
103 KiB
MQL5
1516 lines
103 KiB
MQL5
//+------------------------------------------------------------------+
|
|
//| Topology.mqh |
|
|
//| |
|
|
//| Network bootstrap and topology construction: the derived shape |
|
|
//| (width/taper/depth/conv filters/LSTM hidden), the conv, LSTM and |
|
|
//| batch-norm stages, BuildFreshTopology and InitIndicators. |
|
|
//| |
|
|
//| PARTIAL IMPLEMENTATION FILE - not standalone. |
|
|
//| CExpertSignalAIBase method BODIES only. The class declaration |
|
|
//| lives in Expert\ExpertSignalAIBase.mqh, which includes this file |
|
|
//| at the bottom, after the declaration. Do not include it |
|
|
//| anywhere else and do not compile it on its own. |
|
|
//+------------------------------------------------------------------+
|
|
#ifndef WARRIOR_AIBASE_TOPOLOGY_MQH
|
|
#define WARRIOR_AIBASE_TOPOLOGY_MQH
|
|
//+------------------------------------------------------------------+
|
|
//| Common network bootstrap shared by every AI signal: sets up |
|
|
//| indicators, then loads a saved network or builds a fresh one |
|
|
//| whose only per-signal-type difference is AddCustomLayers(). |
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::InitNeuralNetwork(CIndicators *indicators)
|
|
{
|
|
if(m_isInitialized)
|
|
return true;
|
|
if(indicators == NULL)
|
|
return false;
|
|
m_indicatorsPtr = indicators;
|
|
if(!CExpertSignalCustom::InitIndicators(indicators))
|
|
return false;
|
|
if(!CExpertSignalAIBase::InitIndicators(indicators))
|
|
return false;
|
|
//--- Kick the terminal's async history sync for every cross-asset reference symbol NOW, at init,
|
|
//--- so the ~minute of cross-symbol download runs while the model loads and the label cache
|
|
//--- prebuilds - instead of starting only when the first Build() call finds the symbols unselected
|
|
//--- and the first era (and the one-shot MI report) runs with the panel absent. Non-blocking.
|
|
if(m_useCrossAsset)
|
|
m_crossAsset.Warm((ENUM_TIMEFRAMES)m_period);
|
|
Net = new CNet(NULL);
|
|
if(CheckPointer(Net) == POINTER_INVALID)
|
|
return false;
|
|
//--- Size the first dense layer to the data. HERE and only here: it must be settled before the
|
|
//--- fingerprint below (which hashes it) and must never move afterwards - see ComputeFirstLayerWidth()
|
|
//--- and the note on fingerprint-feeding members at the top of this file. InitIndicators() above is
|
|
//--- what finalises m_neuronsCount, so this is the earliest point the input width is actually known.
|
|
//--- ORDER MATTERS, and it changed on 2026-08-09. The conv and LSTM stages settle FIRST, because
|
|
//--- ComputeFirstLayerWidth() now budgets the dense stack against the width that actually reaches it
|
|
//--- - which, on any topology with a front end, is that stage's output and not the raw input vector.
|
|
//--- Neither of these two depends on m_initialNeuronsCount, so moving them ahead of it is safe;
|
|
//--- depth still comes last because it is derived FROM the first-layer width.
|
|
//--- Both are also sized from the data rather than configured. Unconditional - a plain MLP simply
|
|
//--- never builds the stages these describe, and branching on the topology type would make the
|
|
//--- .cfg contents depend on which subclass is asking.
|
|
//--- The input WINDOW settles first of all: every shape below multiplies by it. Derived for a
|
|
//--- genuinely new model; an existing model ADOPTS the window it was trained with from its .cfg
|
|
//--- further down, exactly like the four shape fields (the .nnw remains the ultimate authority).
|
|
//--- Deliberately after InitIndicators (which needs none of it) and before the fingerprint, whose
|
|
//--- window slot is a LEGACY literal precisely so this measurement cannot re-key a filename.
|
|
m_historyBars = DeriveHistoryBars();
|
|
m_convFilterCount = ComputeConvFilterCount();
|
|
m_lstmHiddenSize = ComputeLstmHiddenSize();
|
|
m_initialNeuronsCount = ComputeFirstLayerWidth();
|
|
//--- Depth LAST of the four: it is derived from the first-layer width above, so it cannot be settled
|
|
//--- before that one is. All four are overwritten from the .cfg further below if this configuration
|
|
//--- already has a trained model - see the adopt-don't-compare block there.
|
|
m_hiddenLayersCount = ComputeHiddenLayerCount();
|
|
//--- The name used to carry a dense-depth tag ("Perceptron 3L"), from when AIType let a user pick
|
|
//--- MLP_3L vs MLP_4L and the depth was the only thing separating two charts of the same family.
|
|
//--- Depth is DERIVED now (see ComputeHiddenLayerCount), so it names nothing anyone chose - it is an
|
|
//--- internal shape detail leaking into a product surface a customer reads. Dropped: the config tag
|
|
//--- appended below ([PAI-0be2]) already disambiguates concurrent charts, and does it correctly for
|
|
//--- every input rather than just this one. The full topology is still logged once at startup by the
|
|
//--- "config -" line, which is where that detail belongs.
|
|
//--- Per-configuration fingerprint appended to the weights filename so that every distinct
|
|
//--- combination of RETRAIN-AFFECTING inputs gets its OWN persistent .nnw/.cfg, instead of all
|
|
//--- combinations sharing one file keyed only on symbol/period/output/optimizer. This is what lets a
|
|
//--- genetic/complete optimization that sweeps network params (neuron counts, layers, reduction,
|
|
//--- history bars, study period, feature set, focal gamma, OOS split, recall/WR targets, ...) build
|
|
//--- each combo's model exactly ONCE and then reuse it on every later pass that revisits that combo -
|
|
//--- previously each differing pass overwrote the single shared cache and retrained from scratch, so
|
|
//--- there was no cross-combination reuse at all. The topology .cfg check further below still runs as
|
|
//--- a secondary guard (and catches a rare hash collision by mismatching and retraining).
|
|
//--- Deliberately covers ONLY params that change the trained weights. Inference-only gates
|
|
//--- (Min_Vote_Open's confidence floor, SignalClusterWindow) and post-training/live settings (money
|
|
//--- management, trailing, entry, filters) are excluded, so changing those still reuses the exact
|
|
//--- same model - matching the pre-existing behavior the optimizer already relied on.
|
|
//--- 2026-08-01: SL/TP LEFT that exempt list. They used to be pure execution settings; the
|
|
//--- triple-barrier relabel makes them the barriers the TARGET is defined by (see TripleBarrierLabel),
|
|
//--- so changing either now changes every label and therefore every weight. A model trained at
|
|
//--- 1:3 must never be silently reused at 1:1. This is the rule from the fingerprint audit applied
|
|
//--- to the newest weight-affecting inputs: what shapes the labels shapes the hash.
|
|
//--- 2026-07-30: every DERIVED value left this hash - first-layer width, dense depth, conv filters,
|
|
//--- LSTM hidden size, and the retired reduction/minNeurons pair. They were legitimately here while
|
|
//--- they were functions of hashed INPUTS, which made them redundant-but-harmless. They stopped being
|
|
//--- that when the capacity budget started measuring the symbol's real bar count: a filename keyed on
|
|
//--- a measured quantity changes the moment more history downloads, so the EA would look for a file
|
|
//--- that does not exist, start from era 0, and orphan a fully-trained model - silently, since a
|
|
//--- missing cache is the normal first-run state and logs as such. The derived shape is pinned in the
|
|
//--- .cfg instead (see the adopt-don't-compare block in LoadAndCompareTopologyConfiguration), which is
|
|
//--- the correct home for it: it describes the model that EXISTS, not the config that asked for it.
|
|
//--- m_studyPeriod is gone for a simpler reason - the input it mirrored no longer exists.
|
|
string fp = StringFormat("%d|%d|%d|%d|%d|%d|%.2f|%d|%d|%d|%d|%d|%d|%d|%d|%d|%d",
|
|
//--- LEGACY_HISTORY_BARS_SLOT: the window left this hash 2026-08-11 when
|
|
//--- it became DERIVED - same rule and reason as every derived field above;
|
|
//--- keyed on a measured quantity, the filename would change the moment more
|
|
//--- history downloads. The .cfg is the record (adopt-don't-compare).
|
|
m_optimizationAlgo, LEGACY_HISTORY_BARS_SLOT, m_outputNeuronsCount,
|
|
m_neuronsCount, m_minTrainYear, LEGACY_CONVERGE_WR_SLOT, m_fractalPeriods,
|
|
//--- LEGACY SLOT (was m_focalGamma, removed 2026-07-31). It was a double fed
|
|
//--- to a %d conversion, so it always contributed the literal below rather
|
|
//--- than the configured gamma - the shipped fingerprints read ...|40|0|30|...
|
|
//--- Writing the same literal keeps every existing model's filename intact.
|
|
m_minDirectionalRecallPct, 0, m_oosSplitPct, m_swingConfirmationBars,
|
|
(int)m_useVolumes, (int)m_useTime, (int)m_useATR, (int)m_useSwingContext,
|
|
(int)m_useNews, m_newsFeatureWindowMinutes);
|
|
//--- The one derived value that DOES belong here, and only when it is not derived at all: a forced
|
|
//--- depth is a developer override (see ForceHiddenLayers), so a build that pins one must not adopt
|
|
//--- the .cfg of a build that derived it. Conditional, so the shipping value of 0 leaves the hash
|
|
//--- exactly as it reads above.
|
|
if(ForceHiddenLayers > 0)
|
|
fp += StringFormat("|FHL:%d", ForceHiddenLayers);
|
|
//--- THE TRIPLE-BARRIER SHAPE LEFT THIS HASH ON 2026-08-07, when SL_Mode/TP_Mode stopped being inputs
|
|
//--- and became MEASURED by ReportBarrierGeometryScan. It is the same rule that moved the horizon out
|
|
//--- (and the derived topology values before it, see above): a filename keyed on a measured quantity
|
|
//--- changes the moment the measurement does - more history downloads, a few more bars shift which
|
|
//--- pairing wins - and the EA then looks for a file that does not exist, starts from era 0, and
|
|
//--- orphans a fully-trained model silently. Measured values are PINNED IN THE .cfg instead, which is
|
|
//--- read back and adopted on load, so a trained model keeps the geometry it was actually trained on.
|
|
//--- Nothing replaces it here on purpose: the .cfg is the record, and re-measuring never happens for a
|
|
//--- model that already exists.
|
|
//--- MA/RSI + AD feature flags appended separately to keep each StringFormat call's arg list modest.
|
|
//--- m_useMA/m_useRSI belong here for the same reason every other feature flag does: they change the
|
|
//--- input-vector width (see InitIndicators()'s m_neuronsCount += 5/+1), so a model trained with them
|
|
//--- on must never silently reuse a cache trained with them off. m_neuronsCount alone (listed above)
|
|
//--- captured the WIDTH but not the composition, so two different feature sets summing to the same
|
|
//--- width could have collided onto one cache file - these two flags close that gap.
|
|
fp += StringFormat("|%d|%d|%d|%d|%d|%d|%d|%d",
|
|
(int)m_useMA, (int)m_useRSI,
|
|
(int)m_useADCumulativeDelta, (int)m_useADShorteningOfThrust,
|
|
(int)m_useADWyckoffEventStream, (int)m_useADWyckoffFailedStructure,
|
|
(int)m_useADWyckoffSignificantBarInversion,
|
|
//--- starting MA TYPE (MA_Type input): changes the MA feature's values, so a change
|
|
//--- must invalidate the cache. The auto-tuned type/period themselves live in the
|
|
//--- .nnw indicator-param block (Flatten/Unflatten), not here - this is the seed only.
|
|
(int)MA_Type);
|
|
//--- MACD/Ichimoku feature flags, appended ONLY WHEN ENABLED rather than unconditionally like every
|
|
//--- flag above. Both spellings are equally correct as a fingerprint (deterministic either way, and a
|
|
//--- model with these on can never collide with one that has them off), but appending them
|
|
//--- unconditionally would have changed the hash of EVERY existing config the moment this feature
|
|
//--- shipped - re-keying and forcing a full retrain of already-converged models that don't use MACD or
|
|
//--- Ichimoku at all. Conditional append leaves those fingerprints byte-identical. The seed periods go
|
|
//--- in for the same reason MA_Type does above: they change the feature's values. Anything added here
|
|
//--- in future should follow the same rule.
|
|
if(m_useMACD)
|
|
fp += StringFormat("|MACD:%d:%d:%d", (int)MACD_PeriodFast, (int)MACD_PeriodSlow, (int)MACD_PeriodSignal);
|
|
if(m_useIchimoku)
|
|
fp += StringFormat("|ICHI:%d:%d:%d", (int)Ichimoku_PeriodTenkan, (int)Ichimoku_PeriodKijun, (int)Ichimoku_PeriodSenkou);
|
|
//--- Cross-asset panel: conditional append, per the rule above, so existing fingerprints are untouched.
|
|
//--- ONLY the flag and the feature count go in. The panel's actual composition - which reference pairs
|
|
//--- were found in Market Watch, and therefore which currencies it can index - is a MEASURED property
|
|
//--- of the terminal, exactly like the bar count the header warns about. Keying the filename on it
|
|
//--- would orphan a fully-trained model the moment the user adds or removes a Market Watch symbol,
|
|
//--- silently, since a missing cache reads as a normal first run. The composition is logged at build
|
|
//--- time and pinned in the .cfg instead.
|
|
if(m_useCrossAsset)
|
|
{
|
|
fp += StringFormat("|XA:%d", CROSSASSET_FEATURES);
|
|
//--- INDEX-MODE RE-ENCODE (2026-08-11). When the traded symbol reports the same currency on
|
|
//--- both sides (SP500 -> USD/USD) the panel's slots changed meaning: denomination + risk-proxy
|
|
//--- strength and a denominator-adjusted divergence, instead of duplicated base/quote series
|
|
//--- (see System\CrossAsset.mqh header). Same width, different SEMANTICS - so models trained
|
|
//--- under the old degenerate encoding must re-key. Appended only for base==quote symbols:
|
|
//--- FX-pair semantics are untouched and their models keep their filenames, per the
|
|
//--- conditional-append rule above. Base/quote is a SYMBOL property, not a measured one, so it
|
|
//--- is fingerprint-safe - it cannot change under a trained model the way Market Watch can.
|
|
if(SymbolInfoString(m_symbol.Name(), SYMBOL_CURRENCY_BASE) ==
|
|
SymbolInfoString(m_symbol.Name(), SYMBOL_CURRENCY_PROFIT))
|
|
fp += ":IDX2";
|
|
}
|
|
//--- Spread feature: conditional append, same rule. Nothing measured goes in - the spread series
|
|
//--- itself is market data, not configuration.
|
|
if(m_useSpreadFeature)
|
|
fp += "|SPR:2";
|
|
//--- Batch normalization changes the LAYER COUNT, not just the weights, so a model trained with it
|
|
//--- must never load into a topology built without it (and vice versa) - the .cfg guard would catch
|
|
//--- the mismatch and retrain, but only after a confusing failure. Appended conditionally, following
|
|
//--- the same rule as MACD/Ichimoku above: a config with batch norm off keeps the fingerprint it
|
|
//--- already had, so shipping this does not re-key and force a retrain of every existing model.
|
|
if(EnableBatchNorm && BatchNormWindow > 1)
|
|
fp += StringFormat("|BN:%d", BatchNormWindow);
|
|
//--- Changes the training gradient, so a model trained with it must never load into a run
|
|
//--- without it. Conditional append, same rule as MACD/Ichimoku/BN above: a config with
|
|
//--- this OFF keeps the fingerprint it already had, so the already-converged models on
|
|
//--- disk stay untouched and remain loadable as the fallback if this regresses.
|
|
if(m_logitAdjustTau > 0.0)
|
|
fp += StringFormat("|LA:%d", (int)MathRound(m_logitAdjustTau * 100.0));
|
|
//--- 2026-07-29 audit of every input in Variables\Inputs.mqh against this hash. Five were changing the
|
|
//--- trained weights without changing the filename, so switching any of them re-adopted a model trained
|
|
//--- under the OLD value - the exact trap that the .nnw architecture incident already cost a day to
|
|
//--- (see EnforceTopologyContract): the .cfg guard would eventually mismatch and retrain, but only
|
|
//--- after a confusing failure, and a matching topology would not mismatch at all.
|
|
//--- LEGACY SLOT. The oversampling/replay inputs this encoded were removed 2026-07-31 (see the
|
|
//--- class-imbalance block in Variables\Inputs.mqh); the replay path itself is gone. The literal is
|
|
//--- the exact string the shipped defaults produced - EnableMinorityReplay=true, OversampleParity=90,
|
|
//--- ConstrainReplay=true - so every model already on disk keeps its filename and stays loadable.
|
|
//--- Dropping the segment instead would re-key EVERY model and force a from-scratch retrain of the
|
|
//--- one topology currently converged and trading, which is a steep price for cosmetics in a hash
|
|
//--- nobody reads. Same treatment as LEGACY_CONVERGE_WR_SLOT / LEGACY_STUDY_PERIOD_SLOT.
|
|
fp += "|MR:1:90:1";
|
|
//--- Feature-value inputs, each conditional on the feature that reads it actually being on - the same
|
|
//--- rule the MACD/Ichimoku blocks above follow. Tick vs real volume feeds different numbers into the
|
|
//--- same input slot (Features.mqh's m_Volumes.Create), and PeriodMA/PeriodRSI seed the tuner
|
|
//--- (ADIndicatorTuner.mqh) exactly as MA_Type does - MA_Type was already hashed, these two were not.
|
|
//--- All three also feed the CLASSIC MA/RSI votes, which are inference-only; gating on the AI feature
|
|
//--- flag is what keeps a classic-signal tweak from re-keying a model that never saw it.
|
|
if(m_useVolumes)
|
|
fp += StringFormat("|VOL:%d", (int)VolumeData);
|
|
if(m_useMA)
|
|
fp += StringFormat("|MAP:%d", (int)PeriodMA);
|
|
if(m_useRSI)
|
|
fp += StringFormat("|RSIP:%d", (int)PeriodRSI);
|
|
//--- AD/WYCKOFF PARAMETERS. These SEED feature values (see CADIndicatorTuner's constructor), so by the
|
|
//--- rule this hash exists to enforce they belong in it - a model trained at one climactic-volume
|
|
//--- threshold must never silently re-adopt weights fitted at another. That rule is not theoretical
|
|
//--- here: the 2026-07-29 audit found five inputs changing trained weights without changing the
|
|
//--- filename, which is the exact trap the .nnw architecture incident already cost a day to.
|
|
//--- APPENDED ONLY ON DEVIATION, following the MACD/Ichimoku/BN/XA convention. All 18 inputs ship at
|
|
//--- byte-identical values to the literals they replaced, so at defaults this token is absent and every
|
|
//--- model already on disk keeps its filename and stays loadable. Without that guard, merely EXPOSING
|
|
//--- these parameters would have re-keyed every config and forced a from-scratch retrain of all four
|
|
//--- topologies - for a change that alters no number anywhere.
|
|
//--- All-or-nothing rather than per-input: one deviation writes the whole vector, so the token is
|
|
//--- either absent or complete and can never encode a partial picture of what the features were built
|
|
//--- from. Gated on the AD features actually being ON, since with them off none of this reaches a
|
|
//--- feature at all (m_useMA/m_useRSI already carry the two shared thresholds' other consumers).
|
|
bool anyAD = (m_useADCumulativeDelta || m_useADShorteningOfThrust || m_useADWyckoffEventStream ||
|
|
m_useADWyckoffFailedStructure || m_useADWyckoffSignificantBarInversion);
|
|
if(anyAD)
|
|
{
|
|
bool adDefaults = (Wyk_VolClimaxMult == WYK_VOL_CLIMAX_DEF &&
|
|
Wyk_VolHighMult == WYK_VOL_HIGH_DEF &&
|
|
Wyk_RangeClimaxMult == WYK_RANGE_CLIMAX_DEF &&
|
|
Wyk_RangeSignificantMult == WYK_RANGE_SIGNIF_DEF &&
|
|
Wyk_ShortTermVolRatio == WYK_ST_VOL_RATIO_DEF &&
|
|
Wyk_AtrMult == WYK_ATR_MULT_DEF &&
|
|
ADCD_Lookback == ADCD_LOOKBACK_DEF &&
|
|
SOT_ThrustLookback == SOT_THRUST_LOOKBACK_DEF &&
|
|
SOT_MinImpulses == SOT_MIN_IMPULSES_DEF &&
|
|
SOT_Threshold == SOT_THRESHOLD_DEF &&
|
|
WES_Lookback == WES_LOOKBACK_DEF &&
|
|
WES_ZigZag == WES_ZIGZAG_DEF &&
|
|
WES_TouchATR == WES_TOUCH_ATR_DEF &&
|
|
WES_ARMinATR == WES_AR_MIN_ATR_DEF &&
|
|
WES_MaxRangeBars == WES_MAX_RANGE_BARS_DEF &&
|
|
WFS_Lookback == WFS_LOOKBACK_DEF &&
|
|
WFS_ZigZagStrength == WFS_ZIGZAG_STRENGTH_DEF &&
|
|
WSBI_Lookback == WSBI_LOOKBACK_DEF);
|
|
if(!adDefaults)
|
|
fp += StringFormat("|ADP:%.4f:%.4f:%.4f:%.4f:%.4f:%.4f:%d:%d:%d:%.4f:%d:%d:%.4f:%.4f:%d:%d:%d:%d",
|
|
Wyk_VolClimaxMult, Wyk_VolHighMult, Wyk_RangeClimaxMult,
|
|
Wyk_RangeSignificantMult, Wyk_ShortTermVolRatio, Wyk_AtrMult,
|
|
(int)ADCD_Lookback, (int)SOT_ThrustLookback, (int)SOT_MinImpulses,
|
|
SOT_Threshold, (int)WES_Lookback, (int)WES_ZigZag,
|
|
WES_TouchATR, WES_ARMinATR, (int)WES_MaxRangeBars,
|
|
(int)WFS_Lookback, (int)WFS_ZigZagStrength, (int)WSBI_Lookback);
|
|
}
|
|
//--- INPUT WINDOW ORDER. Appended UNCONDITIONALLY, which is a deliberate break from the "conditional
|
|
//--- append so existing fingerprints stay byte-identical" rule every block above follows - and the
|
|
//--- reason is exactly why that rule exists in the first place. Every model on disk was trained on a
|
|
//--- window fed NEWEST-BAR-FIRST; BuildFeatureWindow() now feeds it oldest-first (see its definition
|
|
//--- comment for the LSTM measurement that forced it). The vector has the same SHAPE and the same
|
|
//--- features, so nothing downstream would fail: a stale .nnw would load cleanly, pass the .cfg guard,
|
|
//--- and run a model fitted to one input ordering against the other - silently, forever. That is the
|
|
//--- precise failure this hash exists to make impossible, so here re-keying every config is the
|
|
//--- CORRECT outcome, not collateral damage. Version it rather than toggling a flag: if the ordering
|
|
//--- is ever revisited, bump the number instead of trying to reconstruct which models predate what.
|
|
fp += "|WIN:2";
|
|
//--- WYCKOFF CATEGORICAL ENCODING VERSION. Unconditional and versioned for exactly the reason WIN is
|
|
//--- (see above): the 2026-08-09 N1 split re-encodes EventCode/EventPhase/StructuralPhase as
|
|
//--- direction+magnitude pairs, so those input slots now MEAN something different. The vector width
|
|
//--- also changes (13 -> 16 readings), which m_neuronsCount above would catch on its own - but width
|
|
//--- alone is the weaker guard, and a future re-encoding that preserved width would slip past it and
|
|
//--- run a model fitted to one encoding against another, silently and forever. Bump the number
|
|
//--- rather than adding a flag if the encoding is ever revisited.
|
|
//--- Gated on the feature actually being on, so a config without the Wyckoff events keeps its
|
|
//--- existing fingerprint and its trained model.
|
|
if(m_useADWyckoffEventStream)
|
|
fp += "|WES:2";
|
|
//--- TRAINING TARGET (meta-labeling). Conditional, so every existing direction model keeps its
|
|
//--- byte-identical fingerprint. Versioned like WIN/WES: the token covers the meta LABEL and the
|
|
//--- setup-descriptor layout (META_DESC_FEATURES) - bump the number if either ever changes, so a
|
|
//--- model trained on one meaning can never silently load under another. The 2-output head and the
|
|
//--- State\META\ folder already separate the FILES; this separates the SEMANTICS.
|
|
if(IsMetaTarget())
|
|
fp += "|TGT:META1";
|
|
//--- FNV-1a 32-bit -> 8 hex chars: compact, deterministic, order-stable, collision-safe enough for
|
|
//--- the small optimizer grids in play (a collision would merely fail the .cfg guard and retrain).
|
|
uint fpHash = 2166136261;
|
|
int fpLen = StringLen(fp);
|
|
for(int fpi = 0; fpi < fpLen; fpi++)
|
|
{
|
|
fpHash ^= (uint)StringGetCharacter(fp, fpi);
|
|
fpHash *= 16777619;
|
|
}
|
|
m_fileName += "_" + DoubleToString(MathRound(m_outputNeuronsCount)) + "_" + DoubleToString(MathRound(m_optimizationAlgo)) + "_" + StringFormat("%08x", fpHash);
|
|
//--- Finish the display name with the model's short id and the leading 4 hex digits of that same
|
|
//--- fingerprint, so every log line and panel names the model file it belongs to. The dense-depth tag
|
|
//--- added at the top of this function separates MLP_3L from MLP_4L, but NOT two charts that differ by
|
|
//--- anything else - the batch-norm control was 3L on both sides, which put two identical
|
|
//--- "Perceptron 3L" streams in the log the first time this was tried. Any config difference at all
|
|
//--- changes the hash, by construction, so it is the discriminator that cannot go stale as inputs are
|
|
//--- added - but it is NOT unique on its own. The fingerprint deliberately omits the topology TYPE,
|
|
//--- because the file path already separates it (State\CONV\ vs State\LSTM\ vs State\HYB\) and hashing
|
|
//--- a value that is constant within a folder would add no discriminating power while re-keying every
|
|
//--- trained model on disk into a forced retrain. The consequence is that CONV, LSTM and HYBRID at the
|
|
//--- same depth with the same inputs hash IDENTICALLY - a 2026-07-30 deploy came back with three charts
|
|
//--- all tagged [4109]. Their files were never at risk; the TAG was simply unable to do its one job.
|
|
//--- Prefixing m_id restores uniqueness on the display side without touching m_fileName, and the hex
|
|
//--- half still greps straight to the .nnw inside the folder the prefix names.
|
|
string cfgTag = " [" + m_id + "-" + StringSubstr(StringFormat("%08x", fpHash), 0, 4) + "]";
|
|
if(StringFind(ID, cfgTag) < 0)
|
|
ID += cfgTag;
|
|
//--- One self-verifying config line per chart, deliberately NOT gated on VerboseMode. A multi-chart
|
|
//--- comparison is only valid if every chart is identical except the axis under test, and until now
|
|
//--- a drifted setting was invisible: the filename carries a HASH, so two charts that should match
|
|
//--- and do not look merely "different" with no indication of WHICH field moved. Printing the raw
|
|
//--- fingerprint string makes the six lines directly diffable - any accidental divergence in study
|
|
//--- period, feature set, focal gamma or anything else that feeds training shows up as a textual
|
|
//--- difference at startup instead of an unexplained result three hours later.
|
|
Print(ID + ": config - " + IntegerToString(m_hiddenLayersCount) + " dense from " +
|
|
IntegerToString(m_initialNeuronsCount) + " units | batchnorm " +
|
|
((EnableBatchNorm && BatchNormWindow > 1) ? "ON(" + IntegerToString(BatchNormWindow) + ")" : "OFF") +
|
|
//--- "requested", not the bare number: the EFFECTIVE tau is capped against the head's usable
|
|
//--- logit range and cannot be known until the class priors are measured, so printing 1.00 here
|
|
//--- read as the value in force when every chart was actually running 0.35. The real figure is
|
|
//--- logged once per run by ApplyLogitAdjustment().
|
|
//--- The ONE class-imbalance mechanism. There is deliberately no "| replay ON/OFF" beside it any
|
|
//--- more: that field reported a path which had already been dead for the whole shipped
|
|
//--- configuration, which is exactly the kind of line that makes a log look informative while
|
|
//--- describing nothing (see the class-imbalance audit in Variables\Inputs.mqh).
|
|
" | class-imbalance " + (m_logitAdjustTau > 0.0
|
|
? "logit-adjust(tau " + DoubleToString(m_logitAdjustTau, 2) + " requested)" : "OFF") +
|
|
" | input " + IntegerToString((int)m_historyBars * m_neuronsCount) +
|
|
" (" + IntegerToString((int)m_historyBars) + " bars x " + IntegerToString(m_neuronsCount) + ")" +
|
|
//--- The front-end stages are DERIVED (see ComputeConvFilterCount/ComputeLstmHiddenSize), so
|
|
//--- without them this "self-verifying" line verified only half the topology - it printed the
|
|
//--- dense taper while the conv/recurrent stages that actually dominate CONV/LSTM/HYBRID were
|
|
//--- invisible. Shows the width flowing INTO each stage as well as out of it, because the
|
|
//--- interesting failure is a stage that expands rather than compresses.
|
|
FrontEndConfigSummary());
|
|
//--- Kept as its own line and deliberately free of any per-chart prefix INSIDE the string, so the six
|
|
//--- startup lines diff textually against each other. The model file goes on the line below rather than
|
|
//--- here for the same reason: it necessarily differs per topology (it carries the State\<id>\ folder),
|
|
//--- so folding it in would make every fingerprint line differ and destroy the diff.
|
|
Print(ID + ": fingerprint - " + fp);
|
|
//--- The resolved path is DebuggingMode-only: the tag above already names the folder (its m_id half)
|
|
//--- and the file's hash suffix (its hex half), so this line is derivable rather than new information,
|
|
//--- and a third startup line per chart is not worth spending on a user who will never open the file.
|
|
if(DebuggingMode)
|
|
Print(ID + ": model file - " + m_fileName + ".nnw");
|
|
//--- Strategy Tester / optimizer: target a LOCAL (agent-sandboxed, non-FILE_COMMON) cache file
|
|
//--- instead of the shared production weights, so genetic/complete optimization passes on this
|
|
//--- same agent can reuse an already-trained model whenever the topology-relevant inputs
|
|
//--- (neuron counts, layers, history bars, output count, opt algo, study period, ...) are
|
|
//--- unchanged from a previous pass, instead of re-running every training era from scratch each
|
|
//--- pass. The live/manual-chart production .nnw/.cfg under FILE_COMMON are never touched by
|
|
//--- this path, so a backtest can never corrupt or overwrite the deployed live model.
|
|
bool inTesterOrOpt = MQLInfoInteger(MQL_TESTER) || MQLInfoInteger(MQL_OPTIMIZATION) || MQLInfoInteger(MQL_FORWARD);
|
|
m_activeFileName = inTesterOrOpt ? (m_fileName + "_optcache") : m_fileName;
|
|
m_activeFileCommon = !inTesterOrOpt;
|
|
//--- Claim these files before anything reads or writes them, and refuse to start if another chart in
|
|
//--- this terminal already holds them (see AcquireConfigLock). Deliberately placed here: this is the
|
|
//--- first moment the resolved filename - i.e. the config's true identity - is known, and it is still
|
|
//--- ahead of every load, seed and save. Tester/optimizer agents are exempt: each is a separate
|
|
//--- process writing its own sandboxed _optcache, and running one config across many agents in
|
|
//--- parallel is the entire point of an optimization.
|
|
#ifdef WARRIOR_EXPORT_FEATURES
|
|
//--- RESEARCH BUILD: no lock. This binary reads history and writes one CSV - it never trains, never saves
|
|
//--- a model (OnTick returns immediately, so no era ever completes) and therefore has nothing to protect
|
|
//--- against a concurrent chart. Claiming the lock would only make the exporter REFUSE to start whenever
|
|
//--- the config it wants to read is already open on a production chart, which is exactly when it is most
|
|
//--- useful to run.
|
|
#else
|
|
if(!inTesterOrOpt && !AcquireConfigLock())
|
|
return false;
|
|
#endif
|
|
//--- Any Strategy-Tester run - a single backtest OR an optimization pass - runs pure inference on the
|
|
//--- deployed model, never trains. A user optimizing TRADING parameters (SL/TP, filters, MM, ...) wants
|
|
//--- the AI held fixed at the deployed weights so passes are fast and comparable; retraining the net
|
|
//--- per config would be slow and make every pass a different model. AI hyperparameters are tuned on a
|
|
//--- chart (the internal auto-tuner / a real training run), not via MT5 optimization. Training and the
|
|
//--- new online continual-learning step therefore run ONLY on a live chart (see OnlineLearnStep).
|
|
m_inferenceOnly = MQLInfoInteger(MQL_TESTER);
|
|
//--- Seed the agent-local optcache from the deployed production model on the first tester/opt pass.
|
|
//--- Without this, the tester's separate _optcache file starts empty and the run retrains from zero -
|
|
//--- so a buyer who loads a .set and hits "backtest" waits through a full training run instead of a
|
|
//--- backtest of the model they deployed. The optcache shares the production model's exact config
|
|
//--- fingerprint (same m_fileName base), so the copied weights are guaranteed topology-compatible.
|
|
//--- Copies FROM FILE_COMMON (the live/manual-chart model) INTO the agent-local sandbox only; the
|
|
//--- production files are read, never written, so a backtest still can't corrupt the deployed model.
|
|
//--- Re-seeds when the cache is MISSING *or* STALE. Staleness matters because a tester run no longer
|
|
//--- writes this file at all (see PersistWeightsOnShutdown's inference-only skip), so without a
|
|
//--- freshness check the very first seeded copy would be reused forever - meaning the obvious workflow
|
|
//--- "retrain/redeploy on the chart, then backtest" would silently keep testing the OLD model. The
|
|
//--- config fingerprint in the filename can't catch this: retraining changes the WEIGHTS, not the
|
|
//--- topology inputs the fingerprint hashes, so the name stays identical.
|
|
bool cacheMissing = !FileIsExist(m_activeFileName + ".nnw");
|
|
bool cacheStale = false;
|
|
if(inTesterOrOpt && !cacheMissing && FileIsExist(m_fileName + ".nnw", FILE_COMMON))
|
|
{
|
|
datetime prodModified = (datetime)FileGetInteger(m_fileName + ".nnw", FILE_MODIFY_DATE, true);
|
|
datetime cacheModified = (datetime)FileGetInteger(m_activeFileName + ".nnw", FILE_MODIFY_DATE, false);
|
|
//--- both timestamps must be readable before trusting the comparison; a 0 means "couldn't tell",
|
|
//--- and re-seeding on an unreadable timestamp every single pass would be worse than not checking.
|
|
cacheStale = (prodModified > 0 && cacheModified > 0 && prodModified > cacheModified);
|
|
if(cacheStale)
|
|
Print(__FUNCTION__ + ": the deployed model is newer than this agent's cached copy - re-seeding so the backtest runs the CURRENT model, not the previously cached one.");
|
|
}
|
|
if(inTesterOrOpt && (cacheMissing || cacheStale))
|
|
{
|
|
if(FileIsExist(m_fileName + ".nnw", FILE_COMMON))
|
|
{
|
|
//--- The .nnw is the only copy that MUST succeed - retried (see CopyFileWithRetry's declaration
|
|
//--- comment) because a live chart's own atomic Save() can be mid-rename on this exact file.
|
|
//--- Its return value used to be ignored entirely, so a failed copy still logged "seeded tester
|
|
//--- cache..." as if it had worked, and the run silently trained from scratch instead.
|
|
if(CopyFileWithRetry(m_fileName + ".nnw", m_activeFileName + ".nnw"))
|
|
{
|
|
//--- Best-effort sidecars: not retried - losing one just means a cold calibration/shadow-blend
|
|
//--- start rather than a wrong/untrained model, which the .nnw copy above already guards against.
|
|
//--- Still share-aware (CopySharedFile, not FileCopy): the live chart holds these open too, so
|
|
//--- plain FileCopy would fail on them for exactly the same reason it failed on the .nnw.
|
|
if(FileIsExist(m_fileName + ".cfg", FILE_COMMON))
|
|
CopySharedFile(m_fileName + ".cfg", m_activeFileName + ".cfg", false);
|
|
if(FileIsExist(m_fileName + "_shadow.nnw", FILE_COMMON))
|
|
CopySharedFile(m_fileName + "_shadow.nnw", m_activeFileName + "_shadow.nnw", false);
|
|
//--- carry the calibration sidecar into the agent sandbox too, so a seeded backtest calibrates its
|
|
//--- live decisions with the deployed model's priors instead of the un-adjusted cold defaults.
|
|
if(FileIsExist(m_fileName + ".stats", FILE_COMMON))
|
|
CopySharedFile(m_fileName + ".stats", m_activeFileName + ".stats", false);
|
|
Print(__FUNCTION__ + ": seeded tester cache from the deployed production model (" + m_fileName + ") - this run reuses the deployed weights instead of retraining");
|
|
}
|
|
//--- else: CopyFileWithRetry already logged why. Fall through - the Net.Load() below will
|
|
//--- correctly report "no file" and BuildFreshTopology() takes over, same as a genuine first pass.
|
|
}
|
|
else if(m_inferenceOnly)
|
|
//--- Name the exact file (symbol + timeframe + config fingerprint) it looked for: the model is
|
|
//--- keyed on the CHART TIMEFRAME, so the #1 cause of this is running the tester on a different
|
|
//--- timeframe than the model was trained on (e.g. an H4 model, tester set to H1) - which reads
|
|
//--- as "no model" when one exists under a different timeframe. Spelling out the filename makes
|
|
//--- that mismatch obvious instead of looking like the deploy silently failed.
|
|
Print(__FUNCTION__ + ": WARNING - no deployed production model found at '" + m_fileName +
|
|
".nnw' (shared folder) for " + _Symbol + " " + EnumToString((ENUM_TIMEFRAMES)_Period) +
|
|
". A single backtest runs inference only and will NOT train. Most common cause: the tester" +
|
|
" timeframe differs from the one the model was trained on (the filename is keyed on timeframe)." +
|
|
" Otherwise, train this configuration on a chart first, then re-run the backtest.");
|
|
}
|
|
if(!LoadAndCompareTopologyConfiguration(m_activeFileName, m_initialNeuronsCount, m_hiddenLayersCount, m_neuronsReduction, m_minNeuronsCount, m_optimizationAlgo, m_historyBars, m_outputNeuronsCount, m_neuronsCount, m_minTrainYear, m_isInitialized, LEGACY_CONVERGE_WR_SLOT, m_fractalPeriods, m_convFilterCount, m_lstmHiddenSize, m_activeFileCommon))
|
|
{
|
|
// Topology/input params diverged from what produced the saved .nnw (or no .cfg exists yet;
|
|
// for inTesterOrOpt this is also the normal "first pass on this agent" case). The stale .cfg
|
|
// was already deleted on a mismatch, but the .nnw weights themselves are shaped for the OLD
|
|
// topology - loading them into a network built to the NEW shape would corrupt state or crash.
|
|
// Drop the incompatible weights/checkpoint too so the Net.Load() below cleanly misses and
|
|
// BuildFreshTopology() takes over (i.e. this pass pays the training cost once, and the result
|
|
// gets cached below for the NEXT pass to reuse, same as a live topology change would).
|
|
if(FileIsExist(m_activeFileName + ".nnw", m_activeFileCommon ? FILE_COMMON : 0))
|
|
{
|
|
Print(__FUNCTION__ + ": " + m_activeFileName + " - topology/input params changed since last save; discarding incompatible saved weights and starting fresh");
|
|
FileDelete(m_activeFileName + ".nnw", m_activeFileCommon ? FILE_COMMON : 0);
|
|
//--- Reaching here means a TRAINED model was just thrown away, so its drawn signals are stale for
|
|
//--- exactly the same reason ResetWeights() clears them: they would otherwise be restored moments
|
|
//--- later (LoadChartSignals runs at the end of this function) and shown as if they belonged to the
|
|
//--- model about to be trained. The sibling "no .cfg yet" case does not reach here (there are no
|
|
//--- weights to delete), so it is handled separately at the fresh-topology branch below.
|
|
ClearPersistedChartSignals("saved weights discarded - topology/input params changed");
|
|
}
|
|
if(FileIsExist(m_activeFileName + "_ckpt.tmp", m_activeFileCommon ? FILE_COMMON : 0))
|
|
FileDelete(m_activeFileName + "_ckpt.tmp", m_activeFileCommon ? FILE_COMMON : 0);
|
|
// Same reasoning applies to the EMA shadow-weight file (see m_shadowNet's declaration comment) -
|
|
// it's shaped for the OLD topology too, and EnsureShadowNet() has no independent way to detect
|
|
// that mismatch on Load() (CNet::Load() doesn't cross-validate against an expected shape). Drop
|
|
// it so EnsureShadowNet() cleanly misses and re-bootstraps from the fresh Net instead.
|
|
if(FileIsExist(m_activeFileName + "_shadow.nnw", m_activeFileCommon ? FILE_COMMON : 0))
|
|
FileDelete(m_activeFileName + "_shadow.nnw", m_activeFileCommon ? FILE_COMMON : 0);
|
|
//--- the calibration sidecar is tied to the discarded weights - drop it too so a fresh run
|
|
//--- re-measures priors from scratch instead of adjusting with a stale model's base rates.
|
|
if(FileIsExist(m_activeFileName + ".stats", m_activeFileCommon ? FILE_COMMON : 0))
|
|
FileDelete(m_activeFileName + ".stats", m_activeFileCommon ? FILE_COMMON : 0);
|
|
SaveTopologyConfiguration(m_activeFileName, m_initialNeuronsCount, m_hiddenLayersCount, m_neuronsReduction, m_minNeuronsCount, m_optimizationAlgo, m_historyBars, m_outputNeuronsCount, m_neuronsCount, LEGACY_STUDY_PERIOD_SLOT, m_minTrainYear, m_isInitialized, LEGACY_CONVERGE_WR_SLOT, m_fractalPeriods, m_convFilterCount, m_lstmHiddenSize, m_activeFileCommon);
|
|
}
|
|
double loadedIndicatorParams[];
|
|
//--- Inference-only backtest: if this deployed model was validated MQL5-inference-safe at deploy
|
|
//--- (marker in its .stats), load it host-only and run the pure-MQL5 forward path so the backtest
|
|
//--- never loads WarriorDML/WarriorCPU.dll - no DLL file-lock class of failure, and the exact math
|
|
//--- the Market build ships. Falls back to a compute backend just below if that load fails.
|
|
if(m_inferenceOnly && CheckPointer(Net) != POINTER_INVALID)
|
|
{
|
|
LoadModelStats(m_activeFileName, m_activeFileCommon); // reads m_mqlInferenceValidated (and priors)
|
|
if(m_mqlInferenceValidated)
|
|
{
|
|
Net.SetCpuInference(true);
|
|
PrintVerbose(__FUNCTION__ + ": " + ID + " - inference-only backtest running pure-MQL5 (DLL-free): the deployed model is validated MQL5-inference-safe");
|
|
}
|
|
}
|
|
bool netLoaded = LoadNetWithRetry(loadedIndicatorParams);
|
|
//--- Pure-MQL5 load failed unexpectedly (should not happen for a validated model) - drop back to a
|
|
//--- compute backend and retry once so the backtest still runs via the DLL rather than on a fresh net.
|
|
if(!netLoaded && CheckPointer(Net) != POINTER_INVALID && Net.CpuInference())
|
|
{
|
|
Print(__FUNCTION__ + ": " + ID + " - pure-MQL5 load failed; retrying with a compute backend (DLL)");
|
|
Net.SetCpuInference(false);
|
|
netLoaded = LoadNetWithRetry(loadedIndicatorParams);
|
|
}
|
|
//--- the file may carry a superseded architecture - correct it before anything reads the net
|
|
if(netLoaded)
|
|
EnforceTopologyContract();
|
|
//--- A superseded conv receptive field cannot be repaired in place (different weight-tensor shape), so
|
|
//--- the loaded net is discarded and the fresh-topology path below rebuilds and retrains. Deliberately
|
|
//--- routed through netLoaded rather than a separate branch: that path already cools the calibration,
|
|
//--- resets the era/trainingComplete state and re-arms the label-cache prebuild, all of which a genuine
|
|
//--- architecture change needs too.
|
|
if(netLoaded && m_topologySuperseded)
|
|
netLoaded = false;
|
|
//--- restore the calibration sidecar (priors + confidence scale) that pairs with these weights, so a
|
|
//--- restart - including a buyer's inference-only backtest - calibrates live decisions exactly as the
|
|
//--- saved model did instead of running with cold defaults (priors 0 => no adjustment). See LoadModelStats().
|
|
if(netLoaded)
|
|
LoadModelStats(m_activeFileName, m_activeFileCommon);
|
|
m_modelLoadedFromDisk = netLoaded;
|
|
//--- Make a successful resume visible (the counterpart to the fresh-start / mismatch messages below):
|
|
//--- on a live chart this confirms the saved model was found and loaded rather than silently retrained.
|
|
if(netLoaded && !inTesterOrOpt)
|
|
Print(ID + ": resumed saved model from era " + IntegerToString(m_eraCount) + " (trainingComplete=" + (string)m_trainingComplete + ") - continuing, not retraining from era 0.");
|
|
//--- RESUMED MODELS GET THE SAME WARM-UP AS FRESH ONES (2026-08-13; was `netLoaded ? 0 : 3`).
|
|
//--- The old rationale said a restart "already has a proven-synced history" - but the custom
|
|
//--- indicators recompute from scratch every PROCESS start regardless of what the .nnw proves,
|
|
//--- and skipping the warm-up on resume is the exact root cause that has now bitten three times:
|
|
//--- the cold ATR cached as permanent (ba13eef), the cold AD block training on zeros (the
|
|
//--- 2026-08-11 guard), and the 2026-08-13 resumed-META stall (features read milliseconds after
|
|
//--- OnInit while five AD indicators were still calculating 54k bars on a memory-starved box,
|
|
//--- pass 1 hot-looping 0->100% for 6+ minutes). Three no-op passes cost seconds.
|
|
//--- The label cache itself, however, is NEVER restored from the .nnw checkpoint - it lives only in
|
|
//--- the in-memory m_labelCacheBuy/Sell/HasValue arrays, which start empty every process start
|
|
//--- regardless of netLoaded. Previously this was set to netLoaded, which on a successful checkpoint
|
|
//--- load skipped the eager StartLabelCachePrebuild()/AdvanceLabelCachePrebuild() scan (Train()'s
|
|
//--- !m_labelCachePrebuilt gate) - every bar then fell through to the lazy per-bar fallback in the
|
|
//--- era loop, which calls ComputeLabelForBar(), a dead stub that unconditionally returns
|
|
//--- buy=false/sell=false (the real labeling logic lives ONLY in
|
|
//--- AdvanceBarrierLabelState(), reachable exclusively from the eager prebuild). The result: every
|
|
//--- restart that loaded a checkpoint silently force-labeled the entire era Neutral until something
|
|
//--- else (a topology mismatch, a fresh start) triggered a real prebuild. Always eager-prebuilding
|
|
//--- now, checkpoint or not, closes this at the root - the dead stub fallback then never matters.
|
|
m_warmupPassesRemaining = 3;
|
|
m_labelCachePrebuilt = false;
|
|
if(inTesterOrOpt && netLoaded)
|
|
Print(__FUNCTION__ + ": " + ID + " - reused cached weights from a previous optimization/tester pass on this agent (era " + IntegerToString(m_eraCount) + ", trainingComplete=" + (string)m_trainingComplete + ") - skipping redundant training for this unchanged config");
|
|
if(netLoaded && ArraySize(loadedIndicatorParams) == AD_TUNE_PARAM_COUNT)
|
|
{
|
|
// Restart deploying previously AutoTune-d indicator params even with AutoTuneIndicators=false now.
|
|
// AdoptIndicatorParams (NOT a bare ReInitADIndicators): when the saved params equal the values the
|
|
// indicators were JUST created with - the common case, since the MI tuner usually keeps the
|
|
// configured settings - re-creating is a pure destroy/rebuild churn paid at the worst possible
|
|
// moment (milliseconds after process start, history still syncing). Observed 2026-08-13 on a
|
|
// resumed META model: the churn threw away five freshly-calculating indicator instances and the
|
|
// replacements sat cold for 6+ minutes on a memory-starved box, stalling training entirely.
|
|
AdoptIndicatorParams(loadedIndicatorParams, indicators);
|
|
}
|
|
if(!netLoaded)
|
|
{
|
|
int error_code = GetLastError();
|
|
//--- Do NOT present error_code as the cause: on a no-GPU/CPU-DLL box it is the harmless 5100
|
|
//--- (OpenCL-not-found) left by the compute probe inside CNet::Load, NOT the reason the file was
|
|
//--- rejected. CNet::Load now prints the precise reason (bad marker / type mismatch / 0-layer stub /
|
|
//--- partial layer load) itself. Only clear the stale code here; the "rebuilding fresh" line below is
|
|
//--- the user-facing summary.
|
|
if(error_code != 5004) // not "file not found"
|
|
ResetLastError();
|
|
//--- CRITICAL: a failed load may have ALREADY overwritten the training-state out-params from the
|
|
//--- bad file's header before it was rejected - notably a corrupt/empty 0-layer stub whose header
|
|
//--- still says trainingComplete=1 (see CNet::Load's 0-layer guard). Left as-is, the freshly built,
|
|
//--- untrained topology below would be treated as an already-deployed converged model: it would
|
|
//--- never train, run inference on random weights (every bar scores Neutral, so the end-of-era NMS
|
|
//--- sweep deletes every chart arrow), and "save weights" would just re-persist that empty net.
|
|
//--- Force the state back to a genuine fresh start so BuildFreshTopology() actually gets trained.
|
|
m_trainingComplete = false;
|
|
m_eraCount = 0;
|
|
dtStudied = 0;
|
|
dForecast = 0;
|
|
//--- Cold the in-memory calibration so the freshly-rebuilt (untrained) topology below runs with no
|
|
//--- stale prior-correction until a retrain re-measures it (priors 0 => AdjustedSignalFromSoftmax is a
|
|
//--- no-op; scale 1.0 = the constructor default). In-memory ONLY - deliberately does NOT touch any
|
|
//--- file. (Earlier this session I also deleted the .stats/_shadow.nnw sidecars here; that was too
|
|
//--- destructive - a load failure can be transient/spurious (a not-yet-ready compute backend, a
|
|
//--- momentary file lock, or - on this CPU-DLL machine - GetLastError() being polluted with the
|
|
//--- harmless OpenCL-not-found 5100 from the probe inside CNet::Load), and wiping a user's calibration
|
|
//--- and deployed shadow on any such hiccup is the wrong default. The sidecars self-heal anyway: the
|
|
//--- shadow re-blends toward the retrained Net and .stats is overwritten on the next save.)
|
|
m_priorBuy = 0.0;
|
|
m_priorSell = 0.0;
|
|
m_priorNeutral = 0.0;
|
|
m_confidenceCalScale = 1.0;
|
|
//--- Accurate diagnostic (do NOT cite GetLastError() - inside CNet::Load the OpenCL probe leaves 5100
|
|
//--- there on a no-GPU/CPU-DLL box, which has nothing to do with the file). Distinguish an ordinary
|
|
//--- fresh start (no file yet) from a real read failure of an existing file by testing existence.
|
|
if(!inTesterOrOpt)
|
|
{
|
|
int loadFlags = m_activeFileCommon ? FILE_COMMON : 0;
|
|
if(FileIsExist(m_activeFileName + ".nnw", loadFlags))
|
|
Print(ID + ": could not read the existing model file " + m_activeFileName + ".nnw - rebuilding a fresh topology to retrain from era 0. Existing .stats/_shadow.nnw are KEPT (they refresh as training runs). If this recurs, that .nnw is likely corrupt - back it up, then use the panel's reset-weights to start clean.");
|
|
else
|
|
Print(ID + ": no saved model for this config yet - starting a fresh training run from era 0.");
|
|
}
|
|
//--- Re-seed before building a fresh topology so weight init is genuinely random. A prior
|
|
//--- genetic tuner sweep (TuneIndicatorsAndTrain's candidate eval loop, line ~5006) left the
|
|
//--- MQL5 RNG to a fixed seed; if this load-fail path then builds a production
|
|
//--- topology without re-seeding, the deployed model's weights would be deterministic/repeatable
|
|
//--- from whatever the last candidate's seed was — silently reproducible, not genuinely random.
|
|
//--- Matches ResetWeights() and Warrior_EA.mq5's OnInit.
|
|
MathSrand(GetTickCount());
|
|
//--- Era 0 with no weights behind it, so any arrow currently on this chart was drawn by a
|
|
//--- DIFFERENT model - the previous fingerprint's, or a corrupt .nnw's. Neither the panel reset
|
|
//--- nor the topology-mismatch discard above covers this path: both are gated on there being a
|
|
//--- saved .nnw to delete, and here there is none (a changed config produces a new m_fileName,
|
|
//--- so the old model's files are not "discarded", they are simply not this model's files).
|
|
//--- Left alone the stale arrows do NOT just look wrong - the chart objects survive deploys and
|
|
//--- restarts on their own, and SaveChartSignals() rebuilds the sidecar by scanning the chart,
|
|
//--- so the first save of this fresh run would adopt the dead model's calls as its own history.
|
|
//--- Deliberately at this call site rather than inside BuildFreshTopology(): the genetic tuner
|
|
//--- calls that for every throwaway candidate (AutoTune.mqh) and must not touch the chart.
|
|
ClearPersistedChartSignals("fresh topology at era 0 - arrows belong to a previous model");
|
|
if(!BuildFreshTopology())
|
|
return false;
|
|
}
|
|
TempData = new CArrayDouble();
|
|
if(CheckPointer(TempData) == POINTER_INVALID)
|
|
return false;
|
|
if(netLoaded)
|
|
// Populate dPrevSignal from the just-loaded weights immediately, rather than leaving it at
|
|
// its blank constructor default until the next (asynchronous, queued) training pass happens
|
|
// to run - matters most for the tester cache-reuse path above, where training may be skipped
|
|
// entirely for this run because dtStudied already covers the whole backtest window.
|
|
RefreshLatestSignal();
|
|
//--- Status line must match what the gate below (if(!m_trainingComplete && !m_inferenceOnly)) will
|
|
//--- actually do - otherwise an inference-only single backtest logs "resuming full training now" right
|
|
//--- under the "runs inference only and will NOT train" warning, which reads as a contradiction.
|
|
string trainState = m_trainingComplete
|
|
? "already complete - staying converged, no full retrain on this restart"
|
|
: (m_inferenceOnly
|
|
? "NOT complete, but this is an inference-only backtest - NOT training (see warning above); deploy a trained model for meaningful results"
|
|
: "NOT complete (interrupted or never converged) - resuming full training now");
|
|
Print(__FUNCTION__ + ": " + m_activeFileName + " - training " + trainState);
|
|
//--- Only kick off a full Train() run here if the loaded model genuinely isn't converged yet - an
|
|
//--- already-complete model used to get one full era-loop retrain (real Net.backProp() over the
|
|
//--- whole IS window) on every single EA restart/reattach for no reason, since this "Init" event
|
|
//--- bypassed ScheduleTrainingIfNeeded()'s m_trainingComplete gate entirely. dPrevSignal is already
|
|
//--- fresh from RefreshLatestSignal() above; ScheduleTrainingIfNeeded()'s normal per-tick check
|
|
//--- will call RefreshConvergedSignal() itself once a genuinely new bar closes.
|
|
if(!m_trainingComplete && !m_inferenceOnly)
|
|
bEventStudy = EventChartCustom(ChartID(), 1, (long)MathMax(0, MathMin(iTime(_Symbol, PERIOD_CURRENT, (int)(100 * Net.recentAverageSmoothingFactor * (m_trainingComplete ? 1 : 10))), dtStudied)), 0, "Init");
|
|
//--- Restore arrows persisted from a previous session (see SaveChartSignals). MUST run here, not in
|
|
//--- InitIndicators(): the arrows file is keyed on the FULL m_fileName including the per-config
|
|
//--- fingerprint, which is only appended above - see the note left at InitIndicators()'s old call site.
|
|
LoadChartSignals();
|
|
//--- bootstrap (or restore) the EMA shadow net now rather than waiting for the first
|
|
//--- RefreshLatestSignal()/era-blend call to lazily trigger it - see m_shadowNet's declaration
|
|
//--- comment.
|
|
EnsureShadowNet();
|
|
m_isInitialized = true;
|
|
#ifdef WARRIOR_EXPORT_FEATURES
|
|
//--- Research build only. Runs here because this is the first point at which the indicators, the buffers
|
|
//--- and the derived barrier horizon are all settled, and it needs no model, no labels and no training.
|
|
ExportFeatureMatrix();
|
|
#endif
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Shared training-set size estimate - see the declaration comment. |
|
|
//+------------------------------------------------------------------+
|
|
double CExpertSignalAIBase::EstimatedInSampleBars(void) const
|
|
{
|
|
int secs = PeriodSeconds(m_period);
|
|
if(secs <= 0)
|
|
secs = PeriodSeconds(PERIOD_H1);
|
|
double barsPerYear = (SECONDS_PER_YEAR / (double)secs) * MARKET_OPEN_FRACTION;
|
|
double oosKept = (100.0 - (double)m_oosSplitPct) / 100.0;
|
|
//--- MEASURED from the symbol's real history, matching Train()'s window exactly (earliest available
|
|
//--- bar, floored by MinTrainYear) now that training covers everything available rather than a
|
|
//--- configured number of years. There used to be a hard rule against reading Bars() here, and it was
|
|
//--- right for what it guarded: what is downloaded grows over a terminal's lifetime, and a topology
|
|
//--- that silently widens as history fills in would re-key its own weights file and throw away a
|
|
//--- trained model. That hazard is now closed at the other end instead - the derived shape is written
|
|
//--- into the .cfg on first build and ADOPTED, not re-derived, on every subsequent load, and none of
|
|
//--- the derived values feed the weights-filename fingerprint any more. So this is measured once per
|
|
//--- model, at the moment the model is created, and never consulted again for an existing one.
|
|
datetime firstAvailableBar = (datetime)SeriesInfoInteger(_Symbol, m_period, SERIES_FIRSTDATE);
|
|
MqlDateTime floorTime;
|
|
TimeCurrent(floorTime);
|
|
floorTime.year = m_minTrainYear;
|
|
floorTime.mon = 1;
|
|
floorTime.day = 1;
|
|
floorTime.hour = 0;
|
|
floorTime.min = 0;
|
|
floorTime.sec = 0;
|
|
datetime windowStart = StructToTime(floorTime);
|
|
if(firstAvailableBar > windowStart)
|
|
windowStart = firstAvailableBar;
|
|
int available = Bars(_Symbol, m_period, windowStart, TimeCurrent());
|
|
//--- History may not have finished syncing when a chart first attaches, and a model whose capacity was
|
|
//--- pinned from a handful of bars would stay crippled for its whole life - the one failure mode that
|
|
//--- measuring instead of assuming introduces. Fall back to a conservative fixed span rather than
|
|
//--- pinning something absurd, and say so, because the fix (reattach once history has synced) is the
|
|
//--- user's to make and is invisible otherwise.
|
|
if(available < TOPOLOGY_BUDGET_MIN_TRUSTED_BARS)
|
|
{
|
|
Print(ID + ": WARNING - only " + IntegerToString(available) + " bars of " + _Symbol +
|
|
" history are available yet, too few to size the network from. Falling back to a " +
|
|
IntegerToString(TOPOLOGY_BUDGET_FALLBACK_YEARS) + "-year assumption. If this is a fresh" +
|
|
" install, let the terminal finish downloading history and then delete this model's weights" +
|
|
" from the panel so the topology is sized from the real data.");
|
|
return (double)TOPOLOGY_BUDGET_FALLBACK_YEARS * barsPerYear * oosKept;
|
|
}
|
|
return (double)available * oosKept;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Derived input-window length - see the declaration comment. |
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::DeriveHistoryBars(void)
|
|
{
|
|
int availableBars = Bars(_Symbol, m_period);
|
|
int span = (int)MathMin(availableBars - 1, WINDOW_DERIVE_SPAN_BARS);
|
|
if(span < TOPOLOGY_BUDGET_MIN_TRUSTED_BARS)
|
|
{
|
|
Print(ID + ": WARNING - only " + IntegerToString(availableBars) + " bars of " + _Symbol +
|
|
" history are available yet, too few to measure the input window from. Falling back to " +
|
|
IntegerToString(HISTORY_BARS_FALLBACK) + " bars. If this is a fresh install, let history "
|
|
"finish downloading and delete this model's weights so the window is measured from real data.");
|
|
return HISTORY_BARS_FALLBACK;
|
|
}
|
|
//--- newest CLOSED bars only (start 1): the forming bar's extremes are still moving
|
|
double hi[], lo[];
|
|
ArraySetAsSeries(hi, true);
|
|
ArraySetAsSeries(lo, true);
|
|
if(CopyHigh(_Symbol, m_period, 1, span, hi) != span ||
|
|
CopyLow(_Symbol, m_period, 1, span, lo) != span)
|
|
{
|
|
Print(ID + ": WARNING - could not read " + IntegerToString(span) + " bars to measure the input "
|
|
"window; falling back to " + IntegerToString(HISTORY_BARS_FALLBACK) + " bars.");
|
|
return HISTORY_BARS_FALLBACK;
|
|
}
|
|
//--- Swing pivots: strict local extremum against the NEWER side, >= against the older side (the
|
|
//--- standard tie-break so a flat top counts once). Alternation is enforced by keeping the more
|
|
//--- extreme of two same-type candidates - a higher high before any low confirms extends the leg,
|
|
//--- it does not end one. Legs are the bar distances between consecutive ALTERNATING pivots: the
|
|
//--- same "confirmed leg" population the barrier horizon medians, measured here from raw price
|
|
//--- because at init the ZigZag indicator has no data yet.
|
|
int legs[];
|
|
ArrayResize(legs, 0, 256);
|
|
int lastType = 0; // +1 swing high, -1 swing low, 0 none yet
|
|
int lastPivotBar = -1;
|
|
double lastExtreme = 0.0;
|
|
for(int b = span - 1 - WINDOW_SWING_WING; b >= WINDOW_SWING_WING; b--) // oldest -> newest
|
|
{
|
|
bool isHigh = true, isLow = true;
|
|
for(int w = 1; w <= WINDOW_SWING_WING && (isHigh || isLow); w++)
|
|
{
|
|
if(hi[b] <= hi[b - w] || hi[b] < hi[b + w])
|
|
isHigh = false;
|
|
if(lo[b] >= lo[b - w] || lo[b] > lo[b + w])
|
|
isLow = false;
|
|
}
|
|
int type = 0;
|
|
if(isHigh != isLow)
|
|
type = isHigh ? 1 : -1; // a bar that is both is degenerate; skip it
|
|
if(type == 0)
|
|
continue;
|
|
if(type == lastType)
|
|
{
|
|
double x = (type == 1) ? hi[b] : lo[b];
|
|
if((type == 1 && x > lastExtreme) || (type == -1 && x < lastExtreme))
|
|
{
|
|
lastPivotBar = b;
|
|
lastExtreme = x;
|
|
}
|
|
continue;
|
|
}
|
|
if(lastType != 0)
|
|
{
|
|
int n = ArraySize(legs);
|
|
ArrayResize(legs, n + 1, 256);
|
|
legs[n] = lastPivotBar - b; // series indices: newer bar = smaller index
|
|
}
|
|
lastType = type;
|
|
lastPivotBar = b;
|
|
lastExtreme = (type == 1) ? hi[b] : lo[b];
|
|
}
|
|
if(ArraySize(legs) < WINDOW_DERIVE_MIN_LEGS)
|
|
{
|
|
Print(ID + ": WARNING - only " + IntegerToString(ArraySize(legs)) + " confirmed swing legs in " +
|
|
IntegerToString(span) + " bars, too few to trust a median. Falling back to " +
|
|
IntegerToString(HISTORY_BARS_FALLBACK) + " bars.");
|
|
return HISTORY_BARS_FALLBACK;
|
|
}
|
|
ArraySort(legs);
|
|
int median = legs[ArraySize(legs) / 2];
|
|
//--- snap DOWN to the ladder (see LEGACY_HISTORY_BARS_SLOT's comment for floor/cap rationale)
|
|
int ladder[] = {12, 16, 20, 24, 32};
|
|
int window = HISTORY_BARS_FLOOR;
|
|
for(int i = 0; i < ArraySize(ladder); i++)
|
|
if(ladder[i] <= median)
|
|
window = ladder[i];
|
|
PrintFormat("%s: derived input window - %d bars (median confirmed swing leg %d over %d legs in %d "
|
|
"bars, snapped down to the ladder%s). Measured once at model creation and pinned in the "
|
|
".cfg; an existing model adopts its own trained window instead.",
|
|
ID, window, median, ArraySize(legs), span,
|
|
median > 32 ? ", CAPPED at 32 - era time scales with the window" : "");
|
|
return window;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Dense-taper depth - see the declaration comment. |
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ComputeHiddenLayerCount(void) const
|
|
{
|
|
//--- Diagnostic escape hatch (compile-time, see ForceHiddenLayers). Deliberately not an input: this
|
|
//--- exists to run depth comparisons while working on the EA, and a user who picks a depth is
|
|
//--- contradicting the width and taper the code derived around it.
|
|
if(ForceHiddenLayers > 0)
|
|
return (int)MathMax(1, MathMin(MAX_HIDDEN_LAYERS, ForceHiddenLayers));
|
|
//--- Depth follows from the two ENDPOINTS the taper already has to connect - the derived first-layer
|
|
//--- width and the output-tied final hidden width (see BuildFreshTopology's taper block) - by asking
|
|
//--- how many steps it takes to get from one to the other at a sane per-layer compression ratio.
|
|
//--- Picking depth independently of those endpoints is what made it meaningless as an input: at 64
|
|
//--- units tapering to 12, four layers compress by 1.4x per step and five by barely 1.3x, so the extra
|
|
//--- depth buys no additional abstraction and costs a vanishing-gradient risk for nothing.
|
|
int lastHidden = (int)MathMax(HIDDEN_TAPER_OUTPUT_MULTIPLE * m_outputNeuronsCount, HIDDEN_TAPER_MIN_WIDTH);
|
|
lastHidden = (int)MathMin(lastHidden, m_initialNeuronsCount);
|
|
if(lastHidden <= 0 || m_initialNeuronsCount <= lastHidden)
|
|
return MIN_HIDDEN_LAYERS;
|
|
double steps = MathLog((double)m_initialNeuronsCount / (double)lastHidden) / MathLog(HIDDEN_TAPER_TARGET_RATIO);
|
|
int layers = (int)MathRound(steps) + 1; // +1: the first layer IS the starting endpoint, not a step
|
|
return (int)MathMax(MIN_HIDDEN_LAYERS, MathMin(MAX_HIDDEN_LAYERS, layers));
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Conv output-filter count - see the declaration comment. |
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ComputeConvFilterCount(void) const
|
|
{
|
|
//--- AddConvStage sets window = ConvReceptiveFieldBars() * m_neuronsCount and step = m_neuronsCount,
|
|
//--- so each sliding position covers that many BARS of features and the layer is a learned projection
|
|
//--- from the whole window down to this many filters. The meaningful reference point is therefore the
|
|
//--- WINDOW width, not one bar's feature count - budgeting against a single bar was correct only while
|
|
//--- the receptive field was 1, and at RF 3 it under-sized the stage 3x (8 filters for a 63-input
|
|
//--- window, an 8x squeeze, where the rule intends 2x). Same rule as before, applied to what the layer
|
|
//--- actually reads: halve the input. Keeping it tied to the window also means the receptive field and
|
|
//--- the filter count can never drift apart the way they did on 2026-07-31.
|
|
int chosen = (ConvReceptiveFieldBars() * m_neuronsCount) / CONV_COMPRESSION_DIVISOR;
|
|
//--- Snap DOWN to a power-of-two ladder for the same reason the first-layer width does: the target is
|
|
//--- approximate, and a value that moves with every feature toggle would re-key the weights file more
|
|
//--- often than the change in capacity justifies.
|
|
int ladder[] = {4, 8, 16, 32};
|
|
int snapped = CONV_FILTERS_MIN;
|
|
for(int i = 0; i < ArraySize(ladder); i++)
|
|
if(ladder[i] <= chosen)
|
|
snapped = ladder[i];
|
|
return (int)MathMax(CONV_FILTERS_MIN, MathMin(CONV_FILTERS_MAX, snapped));
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Derived front-end stages, for the startup config line. |
|
|
//+------------------------------------------------------------------+
|
|
string CExpertSignalAIBase::FrontEndConfigSummary(void) const
|
|
{
|
|
string s = "";
|
|
//--- conv slides a ConvReceptiveFieldBars()-bar window one bar at a time, emitting m_convFilterCount
|
|
//--- filters per position; the optional channel pool + second conv follow. Reported from the shape
|
|
//--- helpers rather than re-derived, so this line always describes what AddConvStage actually built.
|
|
if(UsesConvStage())
|
|
{
|
|
s += " | conv " + IntegerToString(ConvReceptiveFieldBars()) + " bars x" +
|
|
IntegerToString(m_neuronsCount) + "->" + IntegerToString(m_convFilterCount) +
|
|
" (" + IntegerToString(ConvFirstStagePositions()) + " pos)";
|
|
if(HasSecondConvStage())
|
|
s += " | pool /" + IntegerToString(m_convFilterCount) +
|
|
" | conv2 ->" + IntegerToString(ConvOutputPositions()) + " pos x" +
|
|
IntegerToString(m_convFilterCount) + " = " + IntegerToString(ConvOutputWidth());
|
|
else
|
|
s += " = " + IntegerToString(ConvOutputWidth());
|
|
}
|
|
if(UsesLstmStage())
|
|
s += " | lstm " + IntegerToString(LstmFanIn()) + "->" + IntegerToString(m_lstmHiddenSize);
|
|
//--- The dense stack is budgeted against the RAW input, so on any topology with a front-end it can be
|
|
//--- WIDER than the vector reaching it - a linear fan-out that cannot recover information the
|
|
//--- bottleneck already discarded, only add parameters. Flag it rather than silently reshaping a
|
|
//--- trained topology; see ComputeFirstLayerWidth.
|
|
int frontEndOut = UsesLstmStage() ? m_lstmHiddenSize
|
|
: (UsesConvStage() ? ConvOutputWidth() : 0);
|
|
if(frontEndOut > 0 && m_initialNeuronsCount > frontEndOut)
|
|
s += " | NOTE dense fans out " + IntegerToString(frontEndOut) + "->" +
|
|
IntegerToString(m_initialNeuronsCount);
|
|
return s;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Input width the LSTM block actually receives. |
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::LstmFanIn(void) const
|
|
{
|
|
//--- LSTM-only: the layer sits directly on the input, so it sees the whole flattened vector.
|
|
//--- HYBRID: AddConvStage runs first, so the LSTM sees the CONV FEATURE MAP, not the input. That map
|
|
//--- is position-major - ConvOutputPositions() positions of m_convFilterCount filters each - so the
|
|
//--- LSTM's per-timestep width stays m_convFilterCount (AddLstmStage) and its step count is the
|
|
//--- POSITION count, which the conv chain shrinks below m_historyBars once a multi-bar window and a
|
|
//--- second conv are in play. Hardcoding historyBars here would over-state the fan-in and, worse,
|
|
//--- disagree with the width CNet actually hands the layer.
|
|
//--- Budgeting HYBRID's LSTM against the flattened 420 UNDER-sized it by a full ladder step: the
|
|
//--- quadratic in ComputeLstmHiddenSize is dominated by the inputs term, so overstating the fan-in
|
|
//--- buys a smaller H for no reason.
|
|
//--- Requires m_convFilterCount to be settled first - InitNeuralNetwork orders it that way.
|
|
if(HasConvBeforeLstm())
|
|
return ConvOutputWidth();
|
|
return (int)m_historyBars * m_neuronsCount;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| LSTM recurrent hidden width - see the declaration comment. |
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ComputeLstmHiddenSize(void) const
|
|
{
|
|
//--- The LSTM block's parameter count is EXACTLY 4 * H * (H + inputs + 1) - see
|
|
//--- CNeuronLSTMOCL::SetInputs in AI\Network.mqh - and AddLstmStage feeds it the whole flattened
|
|
//--- input vector, so `inputs` is historyBars x neuronsCount. That makes this stage far and away the
|
|
//--- largest weight block in an LSTM or HYBRID model: at the shipped default of 32 units against a
|
|
//--- 540-wide input it is ~73k weights, more than DOUBLE the entire derived dense taper it feeds.
|
|
//--- It was the one part of the network the capacity budget never covered, which is why deriving the
|
|
//--- dense stack alone did not stop LSTM/HYBRID from being over-parameterized.
|
|
//--- Same budget as ComputeFirstLayerWidth: at most one weight per in-sample bar. Solving
|
|
//--- 4H(H+inputs+1) <= isBars for H is an ordinary quadratic, H = (-b + sqrt(b^2+4c))/2 with
|
|
//--- b = inputs+1 and c = isBars/4.
|
|
//--- PER-TIMESTEP width, not the flattened fan-in. The layer is now a recurrence: one shared gate
|
|
//--- block is applied at every step, so its parameter count is 4H(H + stepWidth + 1) - the whole
|
|
//--- point of weight sharing. Budgeting against the flattened width (420, or 160 behind conv) was
|
|
//--- correct for the old single-timestep layer and is now ~20x too pessimistic, which would starve
|
|
//--- the recurrence of hidden units for no reason.
|
|
//--- Must match what the layer is actually built as: a recurrence sizes its shared gate block on the
|
|
//--- PER-TIMESTEP width, while the single-timestep layer reads the whole flattened vector at once.
|
|
//--- Budgeting one against the other over-parameterizes by ~20x in one direction and starves the
|
|
//--- recurrence in the other. See LSTM_SEQUENCE_MODE.
|
|
int inputs = (LSTM_SEQUENCE_MODE ? (HasConvBeforeLstm() ? m_convFilterCount : m_neuronsCount)
|
|
: LstmFanIn());
|
|
double isBars = EstimatedInSampleBars();
|
|
if(inputs <= 0 || isBars <= 0.0)
|
|
return LSTM_HIDDEN_MIN;
|
|
double b = (double)(inputs + 1);
|
|
double budget = (-b + MathSqrt(b * b + 4.0 * (isBars / 4.0))) / 2.0;
|
|
int ladder[] = {8, 16, 32, 64, 128};
|
|
int snapped = LSTM_HIDDEN_MIN;
|
|
for(int i = 0; i < ArraySize(ladder); i++)
|
|
if((double)ladder[i] <= budget)
|
|
snapped = ladder[i];
|
|
return (int)MathMax(LSTM_HIDDEN_MIN, MathMin(LSTM_HIDDEN_MAX, snapped));
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Capacity budget for the first dense layer - see the declaration. |
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ComputeFirstLayerWidth(void) const
|
|
{
|
|
//--- THE WIDTH THAT ACTUALLY REACHES THE DENSE STACK, not the raw input vector.
|
|
//--- Until 2026-08-09 this budgeted against m_historyBars * m_neuronsCount on every topology, which
|
|
//--- is only the truth for a plain MLP. On CONV/LSTM/HYBRID a front-end stage has already reduced
|
|
//--- the vector by the time the first dense layer sees it - an LSTM hands the taper m_lstmHiddenSize
|
|
//--- values (64 on the shipped SP500 H1 config), not 1,280 - so the budget was charging the dense
|
|
//--- layer for ~20x the fan-in it has. Measured on the deployed .cfg files that day: CONV, LSTM and
|
|
//--- HYBRID had all been pinned at FIRST_LAYER_MIN_WIDTH for their whole lives, with the
|
|
//--- "cannot support a N-wide input" warning below firing on a premise that was not true of them.
|
|
//--- The front-end widths come from the same helpers AddConvStage/AddLstmStage build from, so what is
|
|
//--- budgeted and what is constructed cannot disagree - and InitNeuralNetwork settles both stages
|
|
//--- before calling this (see the ordering note there).
|
|
int frontEndOut = UsesLstmStage() ? m_lstmHiddenSize
|
|
: (UsesConvStage() ? ConvOutputWidth() : 0);
|
|
int inputWidth = (frontEndOut > 0) ? frontEndOut : (int)m_historyBars * m_neuronsCount;
|
|
if(inputWidth <= 0)
|
|
return FIRST_LAYER_MIN_WIDTH;
|
|
double isBars = EstimatedInSampleBars();
|
|
//--- One first-layer weight per in-sample bar. That layer is (inputWidth+1) x width and dominates the
|
|
//--- model, so this is effectively a whole-model capacity budget. One parameter per sample is already
|
|
//--- generous for a signal this weak; it is a ceiling, not a target.
|
|
int budget = (int)(isBars / (double)(inputWidth + 1));
|
|
//--- Snap DOWN to the ladder: the estimate above is approximate, and a value that moves with every
|
|
//--- small change would re-key the weights file for no benefit. Rungs are far enough apart that the
|
|
//--- estimate would have to be wrong by ~2x to land on a different one.
|
|
int ladder[] = {16, 32, 64, 128, 256, 512, 1024};
|
|
int chosen = FIRST_LAYER_MIN_WIDTH;
|
|
for(int i = 0; i < ArraySize(ladder); i++)
|
|
if(ladder[i] <= budget)
|
|
chosen = ladder[i];
|
|
//--- NEVER WIDER THAN THE STAGE FEEDING IT. Charging the dense layer only for its real fan-in makes
|
|
//--- the budget generous on a narrow front end - an LSTM's 64 outputs would allow 256 units - and a
|
|
//--- 64 -> 256 first layer is a linear fan-out that cannot recover information the recurrence already
|
|
//--- discarded, it can only add parameters to overfit with. FrontEndConfigSummary() already calls
|
|
//--- that shape out as a defect when it happens; this stops it happening. The taper below this layer
|
|
//--- then funnels as intended. No effect on a plain MLP, which has no front end to be capped by.
|
|
if(frontEndOut > 0)
|
|
chosen = (int)MathMin(chosen, frontEndOut);
|
|
//--- Budget below the floor means this configuration cannot support even the narrowest usable layer -
|
|
//--- the model will be over-parameterized no matter what is chosen here, and no amount of
|
|
//--- regularization fixes having more weights than examples. Typical cause is a high timeframe
|
|
//--- (D1 over 10 years is under 2,000 bars) or too many features for the history available. Say so:
|
|
//--- the fix is fewer HistoryBars / fewer feature groups / a longer study period, none of which this
|
|
//--- function can choose on the user's behalf.
|
|
if(budget < FIRST_LAYER_MIN_WIDTH)
|
|
Print(ID + ": WARNING - " + IntegerToString((int)isBars) + " estimated in-sample bars cannot support a " +
|
|
IntegerToString(inputWidth) + "-wide " +
|
|
(frontEndOut > 0 ? "vector into the dense stack" : "input") + ". The first layer is being floored at " +
|
|
IntegerToString(FIRST_LAYER_MIN_WIDTH) + " units, which is still roughly " +
|
|
DoubleToString((double)(inputWidth + 1) * FIRST_LAYER_MIN_WIDTH / MathMax(1.0, isBars), 1) +
|
|
" weights per training bar - expect overfitting. Reduce HistoryBars or the feature set," +
|
|
" lengthen the study period, or train on a lower timeframe.");
|
|
return MathMax(FIRST_LAYER_MIN_WIDTH, chosen);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Batch-normalization layer - see the declaration comment. |
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::AddBatchNormStage(CArrayObj *topology, int units)
|
|
{
|
|
if(CheckPointer(topology) == POINTER_INVALID)
|
|
return false;
|
|
//--- Not an error: the input is off, so the topology simply has no normalization layers. Returning
|
|
//--- true keeps every call site a plain `if(!Add...) return false;` with no extra branching.
|
|
if(!EnableBatchNorm)
|
|
return true;
|
|
//--- A window of 1 makes the layer a no-op passthrough (mean==x, variance==0), which is a silently
|
|
//--- useless layer rather than an obviously absent one. Refuse to build it instead.
|
|
if(BatchNormWindow <= 1)
|
|
return true;
|
|
CLayerDescription *desc = new CLayerDescription();
|
|
if(CheckPointer(desc) == POINTER_INVALID)
|
|
return false;
|
|
desc.count = units;
|
|
desc.type = defNeuronBatchNorm;
|
|
desc.batch = BatchNormWindow;
|
|
//--- Identity forward transform. The non-linearity belongs to the dense layer stacked on top of this
|
|
//--- one; normalizing and then squashing in the same step would undo the normalization.
|
|
desc.activation = NONE;
|
|
desc.optimization = (ENUM_OPTIMIZATION)m_optimizationAlgo;
|
|
if(!topology.Add(desc))
|
|
{
|
|
delete desc;
|
|
return false;
|
|
}
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Convolution front-end: conv -> channel pool -> conv. Shared by |
|
|
//| CSignalCONV and CSignalHYBRID - see the declaration comment. |
|
|
//| |
|
|
//| MEMORY LAYOUT, which is what every decision here turns on: |
|
|
//| - The INPUT is bar-major: BufferTempData appends m_neuronsCount |
|
|
//| contiguous features per bar, bars in order. So a flat window of |
|
|
//| k*m_neuronsCount spans exactly k consecutive BARS, and a step |
|
|
//| of m_neuronsCount advances exactly one bar. A multi-bar |
|
|
//| receptive field therefore needs NO kernel change. |
|
|
//| - A CONV OUTPUT is position-major: FeedForwardConv (AI\Network.cl)|
|
|
//| emits matrix_o[out + window_out * i], so one position's |
|
|
//| window_out filter responses are CONTIGUOUS and consecutive |
|
|
//| positions sit window_out apart. |
|
|
//| - Both pool implementations (FeedForwardProof, CPU_FeedForwardProof)
|
|
//| slide FLAT: pos = i*step over `window` CONSECUTIVE elements. |
|
|
//| Over a position-major buffer those neighbours are the FILTERS of |
|
|
//| one position. So a pool here is a max-over-CHANNELS, never a |
|
|
//| pool across time. |
|
|
//| |
|
|
//| That is exactly the NeuroNet_DNG reference contract (see |
|
|
//| references\MQL5\Experts\EDL\Trajectory.mqh layers 2-5, kernels |
|
|
//| byte-identical to ours): conv(window=2, step=1, window_out=4) -> |
|
|
//| pool(window=4, step=4) -> conv -> pool. The pool is tied to the |
|
|
//| filter count, giving a clean non-overlapping channel reduction. |
|
|
//| |
|
|
//| WHY THE POOL WINDOW MUST STAY TIED TO window_out. Shipping |
|
|
//| window=3/step=2 against 16 filters overlapped windows across the |
|
|
//| filter axis and straddled position boundaries, collapsing |
|
|
//| unrelated detectors into whichever fired hardest below every |
|
|
//| learnable layer (CONV sat at ~40% balanced accuracy for 510 eras, |
|
|
//| Sell recall 0%). The opposite error is just as bad: dropping the |
|
|
//| pool but leaving conv at window=step=one bar is a 1x1 conv that |
|
|
//| never mixes across time at all. |
|
|
//| |
|
|
//| We stop one layer short of the reference and do NOT append the |
|
|
//| second pool. A channel pool emits one scalar per position, so a |
|
|
//| trailing pool would hand the dense stack ~18 values for a 420-wide |
|
|
//| input and force it to FAN OUT 18 -> 64 instead of funnelling. The |
|
|
//| reference affords that at window_out=4 against a far smaller |
|
|
//| input; here it is a bottleneck below every learnable layer. The |
|
|
//| full 8-filter map goes to the dense/LSTM stage. |
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::AddConvStage(CArrayObj *topology)
|
|
{
|
|
if(CheckPointer(topology) == POINTER_INVALID)
|
|
return false;
|
|
//--- Stage 1: convolution across CONV_RECEPTIVE_FIELD_BARS bars, advancing one bar at a time.
|
|
CLayerDescription *desc = new CLayerDescription();
|
|
if(CheckPointer(desc) == POINTER_INVALID)
|
|
return false;
|
|
//--- desc.count here is the conv layer's own output-filter count (CNeuronConvOCL::Init's window_out
|
|
//--- param, AI\Network.mqh) - was m_hiddenLayersCount (an unrelated dense-taper-depth setting,
|
|
//--- defaulting to 4), bottlenecking every sliding position to just 4 filters regardless of how wide
|
|
//--- the rest of the network was. See ConvFilterCount's declaration comment (Variables\Inputs.mqh).
|
|
desc.count = m_convFilterCount;
|
|
desc.type = defNeuronConv;
|
|
// PRELU, not TANH: matches what CNeuronConv's CPU path (Network.mqh) has always hardcoded
|
|
// regardless of this setting (its activationFunction() override ignores `activation` entirely) -
|
|
// this used to silently diverge from the GPU/DirectML tier, which DOES honor this field and was
|
|
// therefore actually running tanh instead of the intended PReLU whenever hardware accel was active.
|
|
desc.activation = PRELU;
|
|
desc.optimization = (ENUM_OPTIMIZATION)m_optimizationAlgo;
|
|
//--- The whole point: a window spanning several bars. Guarded because m_historyBars can be small
|
|
//--- enough that a multi-bar window would not fit at all, in which case this degrades to the old
|
|
//--- per-bar projection rather than building a negative-width layer.
|
|
desc.window = ConvReceptiveFieldBars() * m_neuronsCount;
|
|
desc.step = m_neuronsCount;
|
|
if(!topology.Add(desc))
|
|
{
|
|
delete desc;
|
|
return false;
|
|
}
|
|
//--- NO POOL, and no second conv. See CONV_RECEPTIVE_FIELD_BARS' comment for the measurement and the
|
|
//--- reference-kernel reading behind that: the conv emits position-major output and the reference pool is
|
|
//--- a flat contiguous max, so a pool here can only ever reduce ACROSS FILTERS within a position, never
|
|
//--- over time. It threw away 87.5% of this layer's output and starved every non-argmax filter of
|
|
//--- gradient. The second conv was mis-shaped in the same change - its window was counted in raw elements
|
|
//--- while its comment claimed positions, so a "2-position" window actually spanned 2 FILTERS of position
|
|
//--- 0 - and it only existed to consume the pool's output.
|
|
//--- If a deeper hierarchy is wanted later, the correct shape on THIS layout is a strided conv over
|
|
//--- positions: window = k * m_convFilterCount, step = s * m_convFilterCount (both whole numbers of
|
|
//--- positions, which IS contiguous in position-major order), never a pool. Springenberg et al. ICLR 2015.
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Conv chain shape. SINGLE SOURCE OF TRUTH - AddConvStage builds |
|
|
//| from these and LstmFanIn/FrontEndConfigSummary report from them, |
|
|
//| so what is constructed and what is logged cannot drift apart. |
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ConvReceptiveFieldBars(void) const
|
|
{
|
|
//--- Degrade to a per-bar projection rather than build an impossible layer when history is too short
|
|
//--- for a multi-bar window. MathMin against m_historyBars keeps window <= input width.
|
|
int bars = (int)MathMin((int)CONV_RECEPTIVE_FIELD_BARS, (int)m_historyBars);
|
|
return (bars > 0 ? bars : 1);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ConvFirstStagePositions(void) const
|
|
{
|
|
//--- Sliding positions of stage 1: window ConvReceptiveFieldBars() bars, step 1 bar.
|
|
int p = (int)m_historyBars - (ConvReceptiveFieldBars() - 1);
|
|
return (p > 0 ? p : 1);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::HasSecondConvStage(void) const
|
|
{
|
|
//--- Permanently false: the conv chain is ONE true convolution. Kept (rather than deleted along with the
|
|
//--- pool + second conv it used to gate) so ConvOutputPositions/ConvOutputWidth stay the single source of
|
|
//--- truth for the chain's shape and a future strided second stage has one place to switch itself on.
|
|
return false;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ConvOutputPositions(void) const
|
|
{
|
|
int p = ConvFirstStagePositions();
|
|
return (HasSecondConvStage() ? p - (ConvReceptiveFieldBars() - 1) : p);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ConvOutputWidth(void) const
|
|
{
|
|
//--- Total element count reaching whatever is stacked above the conv chain: the conv output is
|
|
//--- position-major, window_out filters per position.
|
|
return ConvOutputPositions() * m_convFilterCount;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| LSTM sequence stage. Shared by CSignalLSTM and CSignalHYBRID - |
|
|
//| see the declaration comment. |
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::AddLstmStage(CArrayObj *topology)
|
|
{
|
|
if(CheckPointer(topology) == POINTER_INVALID)
|
|
return false;
|
|
CLayerDescription *desc = new CLayerDescription();
|
|
if(CheckPointer(desc) == POINTER_INVALID)
|
|
return false;
|
|
desc.count = m_lstmHiddenSize;
|
|
desc.type = defNeuronLSTM;
|
|
desc.activation = TANH;
|
|
//--- CNeuronLSTMOCL now has an accelerated SGD+momentum kernel (LSTM_UpdateWeightsMomentum,
|
|
//--- AI\Network.mqh/Network.cl/DirectML\WarriorCPU.cpp/WarriorDML.cpp) alongside the original
|
|
//--- Adam one, so this layer honors the same TrainingOptimizer input as PAI/CONV - see
|
|
//--- m_optimizationAlgo's declaration comment.
|
|
desc.optimization = (ENUM_OPTIMIZATION)m_optimizationAlgo;
|
|
//--- PER-TIMESTEP input width - the feature count for ONE bar as it reaches this layer. CNet passes
|
|
//--- this to CNeuronLSTMOCL::SetStepWidth(), which is what makes the layer an actual recurrence over
|
|
//--- m_historyBars steps instead of a single gated projection over the whole flattened vector. It
|
|
//--- must divide LstmFanIn() exactly, which it does by construction in both placements: on the raw
|
|
//--- input the vector is historyBars x m_neuronsCount, and behind the conv chain it is
|
|
//--- ConvOutputPositions() x m_convFilterCount (position-major, filters contiguous per position -
|
|
//--- see the layout note above AddConvStage). Note the step COUNT is the position count, which the
|
|
//--- conv chain shrinks below historyBars once a multi-bar window and a second conv are in play.
|
|
//--- 0 disables sequence mode in CNeuronLSTMOCL::SetStepWidth (which maps <=0 to "not a sequence"),
|
|
//--- restoring the single-timestep layer. See LSTM_SEQUENCE_MODE.
|
|
desc.window = (LSTM_SEQUENCE_MODE ? (HasConvBeforeLstm() ? m_convFilterCount : m_neuronsCount) : 0);
|
|
//--- MathMax(1,...) guard taken from the HYBRID copy: the CSignalLSTM copy divided unguarded, so a
|
|
//--- historyBars of 1 produced step 0 there and step 1 here for what is meant to be the same layer.
|
|
desc.step = MathMax(1, (int)m_historyBars / 2);
|
|
if(!topology.Add(desc))
|
|
{
|
|
delete desc;
|
|
return false;
|
|
}
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Builds a fresh, untrained topology into Net - the exact layer |
|
|
//| construction InitNeuralNetwork() used to inline for the |
|
|
//| "no saved .nnw" case; factored out so TuneIndicatorsAndTrain() can|
|
|
//| get a clean-slate Net per trial without touching indicator init. |
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::BuildFreshTopology()
|
|
{
|
|
CArrayObj *Topology = new CArrayObj();
|
|
if(CheckPointer(Topology) == POINTER_INVALID)
|
|
return false;
|
|
//--- Input Layer
|
|
CLayerDescription *desc = new CLayerDescription();
|
|
if(CheckPointer(desc) == POINTER_INVALID)
|
|
{
|
|
delete Topology;
|
|
return false;
|
|
}
|
|
//--- NetInputWidth = the bar window plus (meta target only) the per-candidate setup descriptor
|
|
//--- appended after it - see AppendCandidateFeatures. Zero-delta for every direction model.
|
|
desc.count = NetInputWidth();
|
|
desc.type = defNeuron;
|
|
desc.activation = NONE;
|
|
desc.optimization = (ENUM_OPTIMIZATION)m_optimizationAlgo;
|
|
if(!Topology.Add(desc))
|
|
{
|
|
delete Topology;
|
|
return false;
|
|
}
|
|
//--- neuron-type-specific layers (Conv+Pool, LSTM, or none for a plain perceptron)
|
|
if(!AddCustomLayers(Topology))
|
|
{
|
|
delete Topology;
|
|
return false;
|
|
}
|
|
//--- Hidden Layers, tapering from m_initialNeuronsCount down to m_minNeuronsCount, each preceded by
|
|
//--- a batch-normalization layer (no-op when EnableBatchNorm is off). Placed BETWEEN layers rather
|
|
//--- than inside them because every layer here computes activation(W.x+b) in a single kernel - there
|
|
//--- is no seam between the matmul and the non-linearity to insert anything into. Normalizing the
|
|
//--- previous layer's OUTPUT is the equivalent formulation and is exactly what the NeuroNet_DNG
|
|
//--- reference's own worked example does (input -> BatchNorm -> hidden -> output).
|
|
//--- The first one also normalizes whatever the conv/pool/LSTM stage produced, which is the widest
|
|
//--- unbounded stage in the whole network and the one whose scale drift hurts most.
|
|
//--- GEOMETRIC taper from the derived first-layer width down to a final hidden width tied to the
|
|
//--- output count, spread evenly over however many layers the architecture asks for. This replaces a
|
|
//--- pair of inputs (NeuronsReduction, MinNeuronsCount) that were calibrated when the first layer was
|
|
//--- a hand-picked 500: they produced a genuine 500 -> 150 -> 45 funnel there, but against the derived
|
|
//--- width they degenerate. At 64 units, "keep 30% with a floor of 20" gives 64 -> 20 -> 20 - the
|
|
//--- reduction stops mattering after one step and the "minimum" silently becomes the width of every
|
|
//--- layer but the first. Deriving the ratio from the endpoints keeps the funnel shape correct at any
|
|
//--- width, which is the whole point of having derived the width in the first place.
|
|
int lastHidden = MathMax(HIDDEN_TAPER_OUTPUT_MULTIPLE * m_outputNeuronsCount, HIDDEN_TAPER_MIN_WIDTH);
|
|
//--- Never wider than where the taper starts: a narrow first layer (see the D1 case in
|
|
//--- ComputeFirstLayerWidth) must still funnel DOWN, not fan back out.
|
|
lastHidden = MathMin(lastHidden, m_initialNeuronsCount);
|
|
double taperRatio = (m_hiddenLayersCount > 1)
|
|
? MathPow((double)lastHidden / (double)m_initialNeuronsCount, 1.0 / (double)(m_hiddenLayersCount - 1))
|
|
: 1.0;
|
|
//--- Width of the layer immediately below the next batch-norm layer. Only advisory (CNet sizes each
|
|
//--- batch-norm layer from whatever it actually sits on), but kept honest so the descriptor list
|
|
//--- reads correctly. Seeded with the input width - which is also a lie for CONV/LSTM/HYBRID, where
|
|
//--- the custom stage in between has resized things; that is exactly why CNet does not trust it.
|
|
int prevWidth = (int)(m_historyBars * m_neuronsCount);
|
|
bool result = true;
|
|
for(int i = 0; (i < m_hiddenLayersCount && result); i++)
|
|
{
|
|
int n = (i == 0)
|
|
? m_initialNeuronsCount
|
|
: MathMax(lastHidden, (int)MathRound(m_initialNeuronsCount * MathPow(taperRatio, (double)i)));
|
|
result = (AddBatchNormStage(Topology, prevWidth) && result);
|
|
if(!result)
|
|
break;
|
|
prevWidth = n;
|
|
desc = new CLayerDescription();
|
|
if(CheckPointer(desc) == POINTER_INVALID)
|
|
{
|
|
delete Topology;
|
|
return false;
|
|
}
|
|
desc.count = n;
|
|
desc.type = defNeuron;
|
|
desc.activation = HiddenLayerActivation();
|
|
desc.optimization = (ENUM_OPTIMIZATION)m_optimizationAlgo;
|
|
result = (Topology.Add(desc) && result);
|
|
}
|
|
if(!result)
|
|
{
|
|
delete Topology;
|
|
return false;
|
|
}
|
|
//--- Batch norm immediately before the head. This is the one placement that matters most: it is what
|
|
//--- keeps the logit spread from decaying as the weights below it shrink, and it is the precondition
|
|
//--- for ever running an UNBOUNDED head here (see the 2026-07-28 note on desc.activation below).
|
|
if(!AddBatchNormStage(Topology, prevWidth))
|
|
{
|
|
delete Topology;
|
|
return false;
|
|
}
|
|
//--- Output Layer
|
|
desc = new CLayerDescription();
|
|
if(CheckPointer(desc) == POINTER_INVALID)
|
|
{
|
|
delete Topology;
|
|
return false;
|
|
}
|
|
desc.count = m_outputNeuronsCount;
|
|
desc.type = defNeuron;
|
|
// Never write the activation as a literal here: this line only ever reaches a BRAND-NEW topology, so
|
|
// a change made here never touches an existing .nnw (CNeuronBaseOCL::Save persists the activation and
|
|
// Load restores it). OutputLayerActivation() is the single source of truth and EnforceTopologyContract()
|
|
// re-asserts it after every Load.
|
|
// Regression (1 output): TANH - its [-1,1] range maps straight onto the -1/0/1 Sell/Neutral/Buy
|
|
// convention, with no SIGMOID offset or clipped ReLU half.
|
|
// Classification (3 outputs): SIGMOID, deliberately NOT NONE.
|
|
// The forward head must stay BOUNDED. HiddenLayerActivation() is PRELU, so this is the only bounded
|
|
// stage in the forward path, and two downstream constants are calibrated against that: CLASS_LOGIT_SCALE
|
|
// = 6.0 is a temperature gain sized to stretch [0,1] into a usable logit span (against a free logit it
|
|
// is just a 6x amplifier), and the +-3.0 cold-start bias seed reads "sigmoid(+-3) ~= 0.95/0.05". An
|
|
// unbounded head was tried 2026-07-27 and reverted the next day: it starts at exp(6*3) vs exp(6*-3),
|
|
// saturated softmax makes the gradient input-INDEPENDENT, and all four architectures degenerated to
|
|
// one- or two-class output within two eras. Do NOT unbound the head again without simultaneously
|
|
// setting CLASS_LOGIT_SCALE to 1.0 and the bias magnitude to ~0.5.
|
|
// The BACKWARD pass is not 3 independent sigmoid deltas: CNet::backProp/backPropOCL detect the
|
|
// 3-output case and compute a joint softmax + categorical-cross-entropy gradient (softmax_i - target_i),
|
|
// which is what ties the classes together - raising one probability structurally lowers the other two.
|
|
// ApplyClassificationSoftmax() reproduces exactly that normalization at read time.
|
|
desc.activation = OutputLayerActivation();
|
|
desc.optimization = (ENUM_OPTIMIZATION)m_optimizationAlgo;
|
|
if(!Topology.Add(desc))
|
|
{
|
|
delete Topology;
|
|
return false;
|
|
}
|
|
if(CheckPointer(Net) != POINTER_INVALID)
|
|
delete Net;
|
|
Net = new CNet(Topology);
|
|
delete Topology;
|
|
if(CheckPointer(Net) == POINTER_INVALID)
|
|
return false;
|
|
// A fresh topology invalidates any existing shadow (see m_shadowNet's declaration comment) -
|
|
// its weights, if any, are shaped for the OLD Net and would either mismatch dimensionally or,
|
|
// worse, silently blend unrelated weight spaces if the shape happens to coincide. Reset to NULL
|
|
// here; EnsureShadowNet() lazily re-bootstraps a fresh clone of the new Net on first use.
|
|
if(CheckPointer(m_shadowNet) != POINTER_INVALID)
|
|
{
|
|
delete m_shadowNet;
|
|
m_shadowNet = NULL;
|
|
}
|
|
//--- Let EnsureShadowNet() re-attempt the clone bootstrap once for this new topology (see the latch's
|
|
//--- declaration comment) - the old shadow, and any prior failed-bootstrap verdict, no longer apply.
|
|
m_shadowBootstrapAttempted = false;
|
|
//--- A brand-new untrained net has NO online continual-learning history (see OnlineLearnStep): reset
|
|
//--- the watermark/guardrail/counters so a fresh start or a ResetWeights()-then-retrain never resumes
|
|
//--- from a superseded model's learned-up-to point or its stale rolling accuracy. Restored (not reset)
|
|
//--- on a normal reload of an existing model - that path loads them via LoadModelStats() and never
|
|
//--- calls BuildFreshTopology(). Harmless during the post-tune rebuild (online learning is
|
|
//--- gated off there, and the deployed final retrain rebuilds and resets again before deployment).
|
|
m_onlineLearnedUpToTime = 0;
|
|
m_onlineRollingAcc = -1.0;
|
|
m_onlineSamples = 0;
|
|
m_onlineBarsSincePersist = 0;
|
|
m_onlineBlendFrozen = false;
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| |
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::InitIndicators(CIndicators *indicators)
|
|
{
|
|
//--- Reset only the status label on (re-)init; deliberately do NOT PurgeChart() here so previously drawn
|
|
//--- signal arrows survive an EA re-init (recompile / param change / timeframe switch) instead of
|
|
//--- vanishing every time - see SIG_ARROW_PREFIX. Full cleanup still happens in the destructor.
|
|
ClearStatusLabel();
|
|
//--- NOTE: LoadChartSignals() is deliberately NOT called here any more. This method runs from
|
|
//--- InitNeuralNetwork() BEFORE the per-config fingerprint is appended to m_fileName, so at this point
|
|
//--- m_fileName is only "<folder>\<symbol>_<period>" - the load looked for e.g. "SP500_16385.arrows"
|
|
//--- while SaveChartSignals() (which only ever runs post-init, with the finished name) had written
|
|
//--- "SP500_16385_3.00000000_1.00000000_<hash>.arrows". The mismatch made the restore silently no-op on
|
|
//--- every restart from the moment the fingerprint was introduced. It is now called at the END of
|
|
//--- InitNeuralNetwork(), once m_fileName is final. Same family as the fingerprint trap documented at
|
|
//--- BuildConfigFingerprint: anything keyed on m_fileName must run AFTER it is fully built.
|
|
if(!InitOpen(indicators))
|
|
return false;
|
|
if(!InitClose(indicators))
|
|
return false;
|
|
if(!InitLow(indicators))
|
|
return false;
|
|
if(!InitHigh(indicators))
|
|
return false;
|
|
//--- label source, always created unconditionally, same as the OHLC indicators above - see
|
|
//--- m_ADZigZag's declaration comment. Optionally ALSO read as an input feature (m_useSwingContext,
|
|
//--- below) using the same already-running indicator instance - no separate init needed for that.
|
|
if(!InitADZigZag(indicators))
|
|
return false;
|
|
m_neuronsCount = 4; // (close-open)/atr, (high-open)/atr, (low-open)/atr, bullish/bearish flag
|
|
if(m_useVolumes)
|
|
{
|
|
// change ratio, level vs 50-bar baseline, absorption (range per unit volume), volume x range -
|
|
// see BufferTempDataCompute()'s matching block, and research/test_volume.py for the measurement
|
|
// that justified widening this from 1. m_neuronsCount is already in the config fingerprint, so
|
|
// this re-keys existing caches on its own: correct, the input vector genuinely changed shape.
|
|
m_neuronsCount += 4;
|
|
if(!InitVolumes(indicators))
|
|
return false;
|
|
}
|
|
// Unconditional, same reasoning as m_ATR/m_ADZigZag below: m_Time.GetData() is read
|
|
// unconditionally elsewhere (label-eligibility gate, cache anchor, online-learning watermark,
|
|
// arrow timestamps) regardless of whether the cyclical time-of-day/day-of-week values are also
|
|
// opted into as an explicit feature via m_useTime - so the indicator itself must always exist.
|
|
if(!InitTime(indicators))
|
|
return false;
|
|
if(m_useTime)
|
|
{
|
|
m_neuronsCount += 6;
|
|
}
|
|
if(m_useATR)
|
|
{
|
|
//already init in the base class
|
|
m_neuronsCount++;
|
|
}
|
|
if(m_useMA)
|
|
{
|
|
if(!InitMA(indicators))
|
|
return false;
|
|
m_neuronsCount += 5; // (open-MA)/atr, (high-MA)/atr, (low-MA)/atr, (close-MA)/atr, (MA-MA[1])/atr
|
|
}
|
|
if(m_useRSI)
|
|
{
|
|
if(!InitRSI(indicators))
|
|
return false;
|
|
m_neuronsCount++; // RSI/100
|
|
}
|
|
if(m_useMACD)
|
|
{
|
|
if(!InitMACDFeature(indicators))
|
|
return false;
|
|
m_neuronsCount += 3; // main/atr, signal/atr, histogram/atr
|
|
}
|
|
if(m_useIchimoku)
|
|
{
|
|
if(!InitIchimoku(indicators))
|
|
return false;
|
|
// (close-Tenkan)/atr, (close-Kijun)/atr, (Tenkan-Kijun)/atr, (close-SpanA)/atr, (close-SpanB)/atr,
|
|
// signed cloud thickness at this bar, signed PROJECTED cloud thickness, Chikou displacement
|
|
m_neuronsCount += 8;
|
|
}
|
|
if(m_useSwingContext)
|
|
m_neuronsCount += 9; // 5 confirmed-pivot features (direction, distance-since-pivot, prior-leg magnitude, retracement ratio, bars-since-pivot) + 4 recent-context features (Donchian pos 20/50, 20-bar return, 20-bar SMA extension) - see BufferTempDataCompute()'s matching block
|
|
if(m_useNews)
|
|
m_neuronsCount += 2; // NewsRecency, NewsProximity - see BufferTempDataCompute()'s matching block
|
|
if(m_useSpreadFeature)
|
|
m_neuronsCount += 2; // spread/ATR (volatility-regime reading), spread change ratio
|
|
// - see BufferTempDataCompute()'s matching block
|
|
if(m_useCrossAsset)
|
|
m_neuronsCount += CROSSASSET_FEATURES; // FX: base/quote strength + divergence; index: denom/risk-proxy strength
|
|
// divergence, cross-sectional dispersion - System\CrossAsset.mqh
|
|
if(m_useADCumulativeDelta)
|
|
{
|
|
if(!InitADCumulativeDelta(indicators))
|
|
return false;
|
|
m_neuronsCount += 6; // Pressure, CumulativeDelta, BullishPressure, BearishPressure, Absorption, Initiative
|
|
}
|
|
if(m_useADShorteningOfThrust)
|
|
{
|
|
if(!InitADShorteningOfThrust(indicators))
|
|
return false;
|
|
m_neuronsCount += 4; // SOT, SOTEffortRegime, SOTConfirmation, SOTPushRegime
|
|
}
|
|
if(m_useADWyckoffEventStream)
|
|
{
|
|
if(!InitADWyckoffEventStream(indicators))
|
|
return false;
|
|
// 16 = 13 buffers - EventPrice (buffer 4, excluded; see BufferTempDataCompute()'s comment) with
|
|
// THREE of them split into a direction/magnitude pair each: EventCode, EventPhase and
|
|
// StructuralPhase are signed categoricals, so 13 readings now occupy 16 inputs (2026-08-09 audit,
|
|
// N1 - see the split in BufferTempDataCompute for why). No reading was added or dropped.
|
|
m_neuronsCount += 16; // eventDir, eventStage, livePhaseDir, livePhaseMag, ZoneTop, ZoneBottom, structDir, structMag, CHoCHTrendToRange, CHoCHRangeToTrend, SlopeAccumulationBullish, SlopeAccumulationBearish, SlopeDistributionBullish, SlopeDistributionBearish, Reaccumulation, Redistribution
|
|
}
|
|
if(m_useADWyckoffFailedStructure)
|
|
{
|
|
if(!InitADWyckoffFailedStructure(indicators))
|
|
return false;
|
|
m_neuronsCount += 5; // Value, BullishStructuralFailure, BearishStructuralFailure, FailedAccumulation, FailedDistribution
|
|
}
|
|
if(m_useADWyckoffSignificantBarInversion)
|
|
{
|
|
if(!InitADWyckoffSignificantBarInversion(indicators))
|
|
return false;
|
|
m_neuronsCount += 5; // SignificantBarQuality, BullishSignificantBar, BearishSignificantBar, BullishControlFlip, BearishControlFlip
|
|
}
|
|
if(!FolderCreate(m_folderPath, FILE_COMMON))
|
|
{
|
|
if(GetLastError() != 5010) // If the error is not because the folder already exists
|
|
{
|
|
Print("Failed to create folder: " + m_folderPath);
|
|
}
|
|
else
|
|
{
|
|
ResetLastError(); // Reset the error code
|
|
}
|
|
}
|
|
return true;
|
|
}
|
|
#endif
|