koros0111/Warrior_EA
1
0
Fork 0
forked from animatedread/Warrior_EA
Code Pull requests Activity
Warrior_EA/Expert/ExpertSignalAIBase.mqh
AnimateDread e8452913c0 feat: add swing-context feature with confirmed zigzag pivot 
Introduce m_useSwingContext flag and FindConfirmedZigZagPivot method to compute normalized swing direction/magnitude/age features from the existing ADZigZag indicator. Only pivots that are at least m_swingConfirmationBars old are trusted, preventing lookahead bias. The SWING_SCAN_CAP_BARS macro limits backward scan depth. Default is off.
2026-07-19 11:04:38 -04:00

4349 lines
247 KiB
MQL5
Raw Permalink Blame History

//+------------------------------------------------------------------+
//| Warrior_EA |
//| AnimateDread |
//| |
//+------------------------------------------------------------------+
#include "ExpertSignalCustom.mqh"
#include "..\AI\Network.mqh"
#include "..\Variables\IndicatorTuneRanges.mqh"
#include "..\System\StatusLabel.mqh"
#include "..\System\NewsRelevance.mqh"
#include "ADIndicatorTuner.mqh"
//--- Hard 0/1 one-hot targets for the 3-neuron classification head - matches the book's
//--- (nnbook.txt sec. 1.4 "Cross-entropy") cross-entropy formulation, which defines the reference
//--- distribution's occurring-event probability as exactly 1.0 (missing event exactly 0.0), no
//--- smoothing. This used to be softened to 0.9/0.1 (see git history) as a workaround for the OLD
//--- per-neuron independent-sigmoid backward gradient: since that gradient was each neuron's own
//--- raw (target-sigmoid_output) delta with nothing else driving it to zero, a literal 1.0/0.0
//--- target was only ever approached asymptotically, so weights kept growing (up to the MAX_WEIGHT
//--- clamp) chasing it forever. Now that CNet::backProp()/backPropOCL() (AI\Network.mqh) compute a
//--- joint softmax+categorical-cross-entropy gradient (softmax_i - target_i) across all 3 neurons
//--- instead, the winning class's SOFTMAX probability - not any one neuron's raw sigmoid value -
//--- is what needs to approach the target, and softmax can reach very close to 1.0 for the winning
//--- class from ordinary (non-saturated) logit separation, so the old runaway-weight failure mode
//--- is no longer expected to require smoothed targets to avoid. If it resurfaces in practice (the
//--- per-neuron SIGMOID forward pass can still individually saturate before softmax normalizes),
//--- that's the first thing to check before reintroducing smoothing.
#define LABEL_SMOOTH_HIGH 1.0
#define LABEL_SMOOTH_LOW 0.0
//--- Outer safety cap on how many times a single minority-class (Buy/Sell) bar can be duplicated into
//--- m_isTrainQueue - see that queueing block's comment for the full rationale (data-level oversampling
//--- to work around Adam's near-invariance to constant loss reweighting, per Kingma & Ba 2015). The
//--- queueing block derives the REAL per-bar repCount from the RAW previous-era class ratio (uncapped),
//--- bounded by this constant, and applies NO additional per-occurrence loss weight on top of it -
//--- duplication alone carries the whole correction. Two earlier versions of this both stacked a second
//--- multiplicative correction (m_maxClassSampleWeight) on top of repCount and both collapsed in
//--- opposite directions - see the queueing block's comment for the full history. This constant is now
//--- the ONLY thing bounding the correction's strength, so it directly trades off correction speed
//--- against the momentum-correlation risk of a single bar recurring too often in one era's shuffled
//--- replay order. Was 3, which left a real observed ~5.2-5.3x Buy/Sell imbalance only partially
//--- corrected (repCount=3 -> Neutral still ~1.75-1.8x more frequent than each minority class in the
//--- queue) - the network stayed inside the same Neutral-collapse basin the old loss-weight-only
//--- mechanism plateaued in for eras 20-115 (~73.6% OOS, 0% Buy/Sell recall), just reaching that
//--- plateau faster. Raised to 5 to let repCount reach much closer to full class parity for that
//--- observed ratio, per Buda, Maki & Mazurowski 2018 (oversampling to parity is competitive-or-better
//--- and doesn't meaningfully overfit) - the Fisher-Yates shuffle already scatters duplicates across
//--- the whole era's replay order, so the momentum-correlation risk this cap guards against scales with
//--- how clustered repeats land, not with the raw repCount number itself.
#define MAX_OVERSAMPLE_REPLICAS 5
//--- Buy/Sell ground truth is the single exact bar where AdvanceZigZagLabelState() confirms a ZigZag
//--- pivot - on real data that's ~2-3% of bars per class (everything else, including the bar directly
//--- next to a real reversal, is Neutral). Observed in practice as OOS Buy/Sell recall stuck at 0-3%
//--- across many real training eras despite class-balanced loss weighting, label smoothing, and a
//--- bounded output activation all correctly doing their jobs - reweighting a rare example's gradient
//--- makes the optimizer try harder to fit that one bar, it can't manufacture separability between the
//--- pivot bar and its near-identical neighbor if the feature window genuinely can't tell them apart.
//--- AdvanceLabelCachePrebuild() widens each confirmed pivot into a +-LABEL_WINDOW_BARS neighborhood
//--- (never overwriting a DIFFERENT genuine pivot's own exact label) after the sequential ZigZag scan
//--- resolves, so the network is trained/scored against "was this bar near a real reversal" instead of
//--- "was this bar the exact reversal tick." Set to 0 to fall back to the old exact-bar-only behavior.
#define LABEL_WINDOW_BARS 2
//--- Reduce-on-regression learning-rate decay. `eta` (AI\Network.mqh) is a plain global read fresh
//--- by every weight-update call on every backend (native/OpenCL/CPU-DLL/DirectML-DLL alike), so
//--- shrinking it here takes effect on the very next backProp() everywhere at once. A fixed step
//--- size that is large enough to escape a bad random init quickly (see the sharp era 5->14 OOS
//--- accuracy climb this was tuned against) is, by the same token, large enough to overshoot once
//--- training gets close to a good solution - the era 14->16 regression right after that climb is
//--- the classic signature of that overshoot, not a structural bug. The existing best-checkpoint
//--- mechanism just below (m_bestOosForecast/SaveCheckpoint) already guarantees the FINAL deployed
//--- model can't regress; this constant lets the live training process itself settle down instead
//--- of continuing to oscillate around a good solution once it finds one.
#define ETA_DECAY_REGRESSION_PCT 5.0 // only decay after a real regression, not per-era noise
#define ETA_DECAY_FACTOR 0.7
#define ETA_MIN 0.0001
//--- Minimum true OOS samples a class needs this era before its recall is trusted as a real pass -
//--- see directionalRecallOK's declaration comment for the era-44-46 false-convergence this prevents.
//--- 10 is a low bar (still lets a genuinely thin early-run OOS window fall back to "not blocking"
//--- via recallPct==-1), just enough to rule out the zero/near-zero-sample degenerate case.
#define MIN_OOS_CLASS_SAMPLES_FOR_GATE 10
//--- Bound on FindConfirmedZigZagPivot()'s backward scan (m_useSwingContext feature block) - ADZigZag's
//--- own stock defaults (Depth=12) produce pivots frequently enough in normal conditions that this cap
//--- should rarely bind, but a long, unusually strong single-direction run could genuinely go this long
//--- without a confirmed opposite-type pivot. Generous rather than tight: the scan is cheap (plain array
//--- reads, no indicator recompute) and its result is cached per-bar by BufferTempData()'s feature cache,
//--- so the one-time cost of a long scan is paid at most once per unique bar, not per training pass.
#define SWING_SCAN_CAP_BARS 750
//--- EMA shadow-weight deployment blend rate - see m_shadowNet's declaration comment for the full
//--- rationale. 0.01 matches the Tau range (0.001-0.01) used for target-network soft updates in
//--- Dmitriy Gizlyk's reference NeuroNet_DNG-based RL algorithms (references\MQL5\Experts\*\Study.mq5) -
//--- small enough that no single era's raw weights can move the deployed model far, large enough that
//--- the shadow still tracks real, sustained learning within a few dozen eras rather than lagging
//--- forever.
#define SHADOW_WEIGHT_TAU 0.01
//--- Free function, not a class method: needed by CExpertSignalAIBase's constructor init list for
//--- both m_modelEta and m_etaCeiling (see their declaration comments), which runs before member-init
//--- order could safely make one depend on another - this only touches the TrainingOptimizer input
//--- (Variables\Inputs.mqh) and the SgdLearningRate/AdamLearningRate inputs (AI\Network.mqh's `lr`
//--- macro resolves to AdamLearningRate), never class members.
double InitialEtaForOptimizer(void)
{
return (TrainingOptimizer == SGD) ? SgdLearningRate : lr;
}
class CExpertSignalAIBase : public CExpertSignalCustom
{
protected:
string ID;
CiOpen m_Open;
CiClose m_Close;
CiHigh m_High;
CiLow m_Low;
CiVolumes m_Volumes;
CiTime m_Time;
//--- custom price-action/volume indicators (CustomIndicators\*.mq5), loaded via iCustom/CiCustom
CiCustom m_ADCumulativeDelta;
CiCustom m_ADShorteningOfThrust;
CiCustom m_ADWyckoffEventStream;
CiCustom m_ADWyckoffFailedStructure;
CiCustom m_ADWyckoffSignificantBarInversion;
//--- ground truth for the training labels (see m_swingConfirmationBars' declaration comment) -
//--- CustomIndicators\ADZigZag.mq5, a renamed/rebranded copy of the stock MQL5 ZigZag indicator,
//--- always created (not gated behind an Enable* input like the AD* feature indicators above,
//--- since it isn't an optional input feature - it IS the label) and always run at its own stock
//--- defaults (Depth=12, Deviation=5, Backstep=3) - never touched by AutoTuneIndicators, since
//--- tuning the ground truth itself alongside the model being scored against it would let a trial
//--- "improve" its OOS score by cherry-picking an easier target rather than a better model.
CiCustom m_ADZigZag;
//--- currently-active tunable input values for each AD indicator (AutoTuneIndicators search space)
//--- plus their own flatten/unflatten/perturb/best-tracking logic - see CADIndicatorTuner's
//--- declaration comment (Expert\ADIndicatorTuner.mqh) for why this is a separate collaborator
//--- rather than loose fields/methods here: it's entirely self-contained (never touches Net,
//--- Train()'s state machine, or anything else in this class), unlike TuneIndicatorsAndTrain()
//--- below, which orchestrates Train()/Net/checkpointing around it and stays here for exactly that
//--- reason. InpContextMode/InpSessionType/InpSessionCount are session choices, not accuracy
//--- knobs, and are hardcoded to the indicators' own defaults in InitAD*() rather than stored/
//--- tuned in the collaborator.
CADIndicatorTuner m_indicatorTuner;
bool m_autoTuneIndicators;
int m_indicatorTuneTrials;
//--- rebuilds only the AD* indicator handles (in place) so ReInit picks up updated param structs
bool ReInitADIndicators(CIndicators *indicators);
//--- builds a fresh, untrained topology into Net (assumes Net is currently NULL); factored out of
//--- InitNeuralNetwork() so TuneIndicatorsAndTrain() can rebuild weights per trial without
//--- re-running indicator init (which would re-Add() the base indicators into indicators a 2nd time)
bool BuildFreshTopology();
//--- CIndicators collection passed into InitNeuralNetwork(), retained so
//--- TuneIndicatorsAndTrain() can call ReInitADIndicators() between trials
CIndicators *m_indicatorsPtr;
CNet *Net;
//--- EMA "shadow" copy of Net, blended a small step (SHADOW_WEIGHT_TAU) toward Net's weights at
//--- the end of every era (see Train()'s era-end block) rather than replaced outright. Live
//--- trading/inference (RefreshLatestSignal()) reads from this, not from Net directly, so any
//--- single era's raw weights - including an Adam overshoot event - can only ever nudge what's
//--- actually deployed by SHADOW_WEIGHT_TAU, not overwrite it wholesale. Bootstrapped as a clone of
//--- Net (via the same Save()/Load() pattern m_simOosNet uses) the first time InitNeuralNetwork()
//--- runs with no prior shadow file, and persisted alongside Net.Save() thereafter. NULL only
//--- during the brief window before that first bootstrap completes - every read site falls back to
//--- Net in that case, never blocks on it.
CNet *m_shadowNet;
CArrayDouble *TempData;
double dError;
double dUndefine;
double dForecast;
double dPrevSignal;
//--- Live-inference alternation gate for LongCondition()/ShortCondition(). Ground truth comes
//--- from AdvanceZigZagLabelState(): a confirmed ZigZag pivot is Sell at a top, Buy at a bottom,
//--- and a real ZigZag can never confirm two same-type pivots back to back - it tracks one pending
//--- high and one pending low, and only emits a new pivot by flipping to the opposite type (see
//--- ADZigZag.mq5). LABEL_WINDOW_BARS then widens each confirmed pivot into a same-direction
//--- neighborhood, but never bleeds into or overwrites an adjacent DIFFERENT pivot's own exact bar
//--- (see AdvanceLabelCachePrebuild()'s widening pass), so this holds at the "streak" level too:
//--- every contiguous run of non-Neutral ground truth is followed - after however many Neutral bars
//--- - by a run of the OPPOSITE direction, never the same one twice in a row. A live Buy directly
//--- after another live Buy, with no Sell in between, is therefore a pattern the model was never
//--- trained to produce - provably a false fire, never a true one - so LongCondition()/
//--- ShortCondition() suppress it to 0 (neutral/abstain) instead of opening a second same-direction
//--- trade before the awaited reversal. Live-only, in-memory, never persisted, and never read by
//--- Train()'s historical scoring/backprop loop (which computes/consumes its own dPrevSignal
//--- separately per historical bar) - same category as SignalMarketDepth.mqh's spread history.
ENUM_SIGNAL m_lastNonNeutralSignal;
datetime dtStudied;
long m_eraCount; // cumulative era counter, persisted in the .nnw so restarts don't look like they reset progress
bool m_trainingComplete; // persisted: true only once Train() converged (objective+stability), not just interrupted
bool bEventStudy;
//--- out-of-sample holdout: share (%) of the study period never trained on, used only to
//--- measure genuine forward accuracy so overfitting shows up in the stats, not just live/OOS trading
int m_oosSplitPct;
double dOosError; // smoothed OOS mismatch rate (0..100), lower is better
double dOosForecast; // smoothed OOS accuracy (0..100)
int m_oosSamples; // count of OOS predictions evaluated this Train() call
//--- per-era counts of the network's own classification of each bar it fed forward (IS+OOS),
//--- reset at the start of every era; surfaced in the status label text so class imbalance
//--- (e.g. the network collapsing to all-Neutral) is visible while training runs
int m_countBuySignals;
int m_countSellSignals;
int m_countNeutralSignals;
//--- per-era counts of the *true* label of every bar fed forward (IS+OOS), reset alongside the
//--- predicted counts above. Used both to display the actual class distribution in the status
//--- label text and, more importantly, to drive IS-sample oversampling in Train() (see backProp calls
//--- below) - rarer classes get replayed more times so gradient descent doesn't just learn to
//--- always call the majority (usually Neutral) class.
int m_trueBuyCount;
int m_trueSellCount;
int m_trueNeutralCount;
//--- snapshot of the class totals above, taken at the end of the PREVIOUS era (see Train()'s
//--- era-reset block) and held fixed for the whole of the current era. The oversampling ratio
//--- below is computed from these frozen counts rather than the live, still-accumulating
//--- m_true*Count values - using the live counts made the ratio order-dependent within a single
//--- era (chronological, oldest-to-newest bar processing means whichever class happens to be
//--- numerically behind at any given moment gets amplified up to 5x, even if that's just a
//--- transient artifact of which regime the era's early bars came from, not the class's true
//--- overall rarity) - a real source of run-to-run oscillation in the predicted class mix. All
//--- zero on era 0, when oversampling simply falls back to 1x (see the reps calc in Train()).
int m_prevEraTrueBuyCount;
int m_prevEraTrueSellCount;
int m_prevEraTrueNeutralCount;
//--- per-era OOS confusion counts, reset each era; used to compute per-class OOS recall (hits/total)
//--- for the status label text and, more importantly, as an additional convergence gate alongside the
//--- blended dOosForecast accuracy - a model that "wins" only by calling everything Neutral will
//--- have high dOosForecast but near-zero Buy/Sell recall, and should NOT be allowed to converge.
int m_oosBuyHits, m_oosBuyTotal;
int m_oosSellHits, m_oosSellTotal;
int m_oosNeutralHits, m_oosNeutralTotal;
//--- same per-era OOS confusion counts as above but keyed by PREDICTED class instead of true class,
//--- i.e. per-class precision (of the bars this era where the model called Sell, how many actually
//--- were Sell?) rather than recall (of the bars that actually were Sell, how many did it catch?).
//--- Recall alone can't distinguish "the model over-fires Sell and happens to also catch enough real
//--- Buys/Neutrals to clear their recall floors" from "the model is genuinely well-calibrated" - a
//--- skewed Predicted-this-era count (see m_countSellSignals) with recall still passing the gate is
//--- exactly that failure mode, and precision is what would expose it.
int m_oosBuyPredicted, m_oosBuyPredictedHits;
int m_oosSellPredicted, m_oosSellPredictedHits;
int m_oosNeutralPredicted, m_oosNeutralPredictedHits;
//--- Confidence calibration: the classification head's forward pass still uses SIGMOID per-neuron
//--- (see BuildFreshTopology()'s SIGMOID comment - bounded activation, avoids logit runaway), but
//--- the BACKWARD pass (CNet::backProp/backPropOCL in AI\Network.mqh) now trains all 3 neurons
//--- jointly against a true softmax+categorical-cross-entropy gradient (softmax_i - target_i), not
//--- 3 independent binary-cross-entropy targets. ApplyClassificationSoftmax() at READ time
//--- reproduces the same normalization the loss was actually trained against, so the value it
//--- returns is now a genuinely loss-consistent probability (still not literature-perfect
//--- calibration - no temperature scaling/Platt scaling has been applied - but no longer
//--- structurally decoupled from what was optimized). SignedAIConfidence()/g_AISignedConfidence
//--- exposes it, and Money\MoneyIntelligent.mqh:AdjustRiskAmount() scales position-sizing risk%
//--- directly off it.
//--- m_oosConfidenceSum accumulates MathAbs(dPrevSignal) (the claimed confidence) over every
//--- classified OOS bar this era; compared against this era's actual OOS accuracy
//--- ((m_oosBuyHits+m_oosSellHits+m_oosNeutralHits)/m_oosSamples) at era-end to derive
//--- m_confidenceCalScale - a single empirical multiplier ("the model claims 80% on average but is
//--- only right 60% of the time -> scale reported confidence by 0.75") applied to the MAGNITUDE
//--- only, never the sign/class decision, in SignedAIConfidence(). EMA-blended across eras (same
//--- smoothing factor as dOosForecast) so one noisy era can't swing it, and clamped to [0.3, 1.5]
//--- so a thin/degenerate OOS window can't drive it to something absurd. Starts at 1.0 (no
//--- correction) until the first era with OOS samples computes a real value.
double m_oosConfidenceSum;
double m_confidenceCalScale;
//--- minimum acceptable OOS recall (%) for the Buy and Sell classes individually before Train() is
//--- allowed to declare convergence; a class with zero OOS samples this era doesn't block (avoids a
//--- deadlock when a given era's OOS window happens to contain no examples of that class)
int m_minDirectionalRecallPct;
//--- Ceiling on the class-balance sampleWeight multiplier (see its computation in Train(), keyed off
//--- m_prevEraTrueBuyCount/Sell/Neutral). MAX_WEIGHT_DELTA (AI\Network.mqh/.cl, DirectML\WarriorCPU.cpp)
//--- bounds how far a SINGLE weight can move in one step, but it does nothing to stop many small
//--- steps in the same direction from accumulating into a large net swing across an era - with real
//--- label counts around Buy 1340 / Sell 1312 / Neutral 7013 the uncapped ratio is ~5.3x on every
//--- single Buy/Sell example, every era. At the old hardcoded 3.0x ceiling that was STILL strong
//--- enough in practice to let one era's Buy gradients overwrite the separability the previous era
//--- had just built for Sell (and vice versa) - the observed anti-correlated Buy/Sell recall whipsaw
//--- (e.g. era 8 Buy:0%/Sell:26% -> era 9 Buy:10%/Sell:57% -> era 10 Buy:1%/Sell:37%), never both
//--- classes improving together. Was tunable (ClassSampleWeight input, CSW_15/1.5x default) as a
//--- loss-level multiplier, but that mechanism proved structurally too weak against Adam's near-
//--- invariance to constant gradient rescaling (Kingma & Ba 2015) - see the m_isTrainQueue queueing
//--- block's oversampling comment for the full history. Class-balance correction now happens
//--- entirely via data-level oversampling (repCount in that queueing block); this member is
//--- currently unread by that path and kept only as a stored/settable value (ClassSampleWeight
//--- input still exists but has no effect) in case a smaller, additive loss-level nudge is ever
//--- reintroduced on top of oversampling.
double m_maxClassSampleWeight;
//--- Focal loss modulating exponent (Lin et al. 2017, "Focal Loss for Dense Object Detection") -
//--- multiplies on top of the class-balance sampleWeight above (3-neuron classification only; the
//--- 1-output regression head's tanh output has no clean "probability of the true class" reading,
//--- so it's left at factor 1.0 there). The class-balance weight above is a fixed per-CLASS
//--- multiplier - it keeps boosting every Buy/Sell example by the same ratio all era, whether the
//--- network is currently 10% or 95% confident on that particular bar, which is exactly what let a
//--- class it had already learned keep getting reinforced long past the point of diminishing
//--- returns (observed in practice: era 95-114 of one run staying Buy-recall 80-100%+ almost the
//--- entire stretch, weights carried over unreset the whole time). Focal loss instead scales EACH
//--- example individually by (1-pt)^gamma, where pt is the network's own current sigmoid output for
//--- that bar's true class neuron: near 1.0 (full weight) while the network is still wrong/unsure
//--- about this specific bar, decaying toward 0.0 once it's already confidently correct - so
//--- gradient keeps redirecting toward whatever the network hasn't learned yet instead of piling
//--- more identical pressure onto a class it already has, regardless of which class that happens to
//--- be. Multiplies the per-slot weight from m_isTrainQueueWeightScale (see that member's comment -
//--- currently always 1.0, class balance being handled by oversampling instead), and since
//--- (1-pt)^gamma <= 1.0 always, it can only ever shrink that weight further, never exceed it.
double m_focalGamma;
//--- Ground truth for the training labels below now comes directly from the real MQL5 ZigZag
//--- indicator (CustomIndicators\ADZigZag.mq5 - a renamed, logic-untouched copy of the stock
//--- ZigZag.mq5, run at its own stock defaults: Depth=12, Deviation=5 points, Backstep=3), not a
//--- hand-rolled fractal/deviation-%/ATR-trend-context approximation of one. Like any ZigZag, its
//--- most recent 1-3 legs are provisional and can still be revised (repainted) as new bars arrive
//--- (see ADZigZag.mq5's own ExtRecalc), so a candidate bar's ADZigZagBuffer value is only trusted
//--- once at least m_swingConfirmationBars MORE bars have closed after it - see
//--- AdvanceZigZagLabelState()'s declaration comment. Until then it stays Neutral.
int m_swingConfirmationBars;
//--- safety valve: Train()'s do-while loop has no other bound on how many eras it will run
//--- before giving up, so a config that can't reach the convergence objective (e.g. too few
//--- swing-confirmed examples for the min recall bar to be reachable) would otherwise loop
//--- forever, permanently keeping the era-progress status label up instead of the normal per-tick
//--- info line and burning CPU nonstop. If the cap is hit, Train() gives up FOR THIS CALL ONLY
//--- (keeps its best checkpointed weights, stays m_trainingComplete=false) and waits out a
//--- cooldown before OnTickHandler is allowed to schedule another attempt.
int m_maxErasPerRun;
int m_trainRetryCooldownSeconds;
datetime m_trainCooldownUntil;
//--- Train() runs its per-bar loop synchronously, and MQL5 is single-threaded per chart - a
//--- multi-minute era would otherwise starve the terminal's chart-event queue for that whole
//--- stretch, including the control panel's own click/drag hit-testing (Panel\ControlPanel.mqh),
//--- which depends entirely on CHARTEVENT_MOUSE_MOVE being delivered promptly. Train() is
//--- therefore chunked: each call does at most ~TRAIN_TIME_BUDGET_MS of work, then returns and
//--- picks back up exactly where it left off on the next call (re-triggered via the same "New
//--- Bar" custom-event scheduling ScheduleTrainingIfNeeded() already used) - the members below
//--- are what makes that resumable across calls.
bool m_trainRunActive; // true: a run (schedule -> convergence/stop) is in progress, possibly spanning many Train() calls
bool m_eraResumePending; // true: yielded mid-bar-loop last call - resume the SAME era, don't start a new one
int m_resumeBars;
int m_resumeTotalIter;
int m_resumeOosCutoff;
int m_resumeBarIndex;
bool m_resumeAddLoop;
//--- Pass 2 of the era loop: bar indices pass 1 (sequential feedForward/scoring) queued as
//--- IS-eligible for backProp, trained on in a freshly shuffled order instead of pass 1's own
//--- fixed chronological (oldest-to-newest) visitation order. Root cause this targets: Adam's
//--- momentum (b1, AI\Network.mqh) has an effective memory window of ~1/(1-b1) steps, which lands
//--- close to LABEL_WINDOW_BARS's ~5-bar contiguous same-class label runs around each ZigZag
//--- pivot - replaying those runs in the SAME order every single era let momentum lock onto
//--- whichever run it was currently passing through, with the network's end-of-era state
//--- disproportionately reflecting whatever it last walked through (recency), not a globally
//--- balanced fit. Symptom observed in practice: IS error climbing era-over-era (0.16->0.55+ over
//--- 25 eras on the same fixed dataset) instead of settling, and OOS Buy/Sell recall whipsawing
//--- between near-0% and 70-100% despite near-equal Buy/Sell label counts. Shuffling is the
//--- standard SGD fix - see the m_isPass2Active declaration below for how it's sequenced against
//--- pass 1 within Train()'s existing resumable-chunk machinery.
int m_isTrainQueue[];
int m_isTrainQueueCount;
//--- Parallel to m_isTrainQueue (same index, kept in lockstep through the Fisher-Yates shuffle
//--- below) - the per-occurrence weight to apply when this queued slot is trained on in pass 2,
//--- decided once at queue time (see the queueing block's oversampling comment) rather than
//--- recomputed in pass 2. Currently always 1.0: class-balance correction is carried entirely by
//--- repCount (how many times a bar was duplicated into the queue), not by any per-occurrence
//--- weight scaling - see the queueing block's comment for why stacking a second correction here
//--- caused two separate training collapses. Kept as a real per-slot array rather than a literal
//--- 1.0 so a future, smaller/safer supplemental weight could be reintroduced without re-touching
//--- the queueing or shuffle code.
double m_isTrainQueueWeightScale[];
//--- Pass 2's cursor into the (already shuffled) m_isTrainQueue - lets pass 2 itself yield/resume
//--- mid-queue under the same TRAIN_TIME_BUDGET_MS chunk budget pass 1 already yields under.
int m_isTrainCursor;
//--- true: pass 1 (sequential) has finished for this era and pass 2 (shuffled backProp) is either
//--- running or has yielded mid-queue - Train() skips straight past pass 1's loop on resume when
//--- this is set. Reset to false only at a fresh era's start (never mid-run).
bool m_isPass2Active;
//--- true: pass 2 has already run to natural completion for this era (m_isPass2Active's own
//--- false state is ambiguous between "not started yet" and "already finished" - both look
//--- identical to a plain `if(!m_isPass2Active)` check). Needed because pass 3 (OOS scoring) can
//--- itself yield/resume across multiple Train() calls same as passes 1/2 do; without this flag,
//--- every resume into an unfinished pass 3 fell through the `if(!m_isPass2Active)` guards on both
//--- pass 1 and pass 2 and re-ran the ENTIRE shuffled queue again (pass 1 itself was a no-op on
//--- resume since its own loop cursor `i` was already exhausted, but pass 2 re-shuffled and replayed
//--- from scratch every single time) - silently multiplying the predicted-class counts (and the
//--- extra, unintended backProp() calls) once per resume, with no bound, for as long as pass 3 kept
//--- needing more than one chunk to finish. Reset to false only at a fresh era's start, alongside
//--- m_isPass2Active.
bool m_isPass2Done;
//--- Pass 3: chronological, OOS-region-only re-walk that happens AFTER pass 2 has actually trained
//--- on this era's IS data - see m_isTrainQueue's declaration comment for why OOS scoring can no
//--- longer just happen inline during pass 1 (that would score every era's OOS window against
//--- weights from BEFORE this era's training, one full era stale - and for era 0 specifically,
//--- against the still-untrained cold-start network, which is why era 0's OOS recall used to show
//--- a meaningless 100% Neutral / 0% Buy / 0% Sell every time). Cursor walks i downward from
//--- m_oosScoreStartIndex to 0, mirroring pass 1's own iteration bounds/order for whichever bars
//--- satisfy isOOS - order matters here (unlike pass 2) since dOosForecast/dOosError are recursive
//--- EMAs over the visitation sequence, not order-independent.
bool m_isPass3Active;
int m_oosScoreIndex;
int m_oosScoreStartIndex;
datetime m_lastBarTime;
//--- This model's own learning-rate trajectory. `eta` (AI\Network.mqh) is a single file-scope
//--- global shared by every CNet in the process - PAI/CONV/LSTM each train independently and
//--- asynchronously (their own Train() calls are interleaved via separate chart-event/timer
//--- scheduling, not synchronized), but all three previously read AND wrote that one global, so
//--- one model's era-end regression/recovery decay/recovery of `eta` (see Train()'s era-end
//--- block) silently changed the learning rate the OTHER two models' very next backProp() call
//--- used too - an unintended coupling between what's supposed to be three independent training
//--- trajectories. Train() now restores `eta` from this member right before touching it, and
//--- saves it back before returning (both at the mid-chunk yield and at the natural end of the
//--- call) - MQL5 gives one chart's EA a single execution thread and each Train() call runs to
//--- completion or to its own yield point before another event is dispatched, so this save/
//--- restore is enough to isolate each model's schedule with no change needed to the neuron-level
//--- or OpenCL/DirectML call signatures that read the global directly.
double m_modelEta;
//--- Ceiling the era-end recovery bump (Train()'s isBetterEra block) restores `eta` toward - used
//--- to be the raw `lr` constant unconditionally, which is only correct for ADAM. SGD's own starting
//--- rate is the separate SgdLearningRate input (AI\Network.mqh) - clamping SGD's recovery bump to
//--- plain `lr` (AdamLearningRate) would silently cap it back down to Adam's ceiling after the
//--- first regression+recovery cycle, undoing the whole point of giving SGD its own configured
//--- rate. Computed once at construction from this model's own m_optimizationAlgo, matching
//--- m_modelEta's per-instance isolation (see that member's comment).
double m_etaCeiling;
int m_erasSinceCooldown; // eras completed since the last cooldown reset - replaces the old per-call-only "erasThisCall"
CArrayDouble m_oosWindow; // run-scoped OOS stability window (used to be a Train()-local CArrayDouble)
double m_bestOosForecast;
//--- whether the era m_bestOosForecast/the checkpoint was taken from also cleared the per-class
//--- directional recall floor (see directionalRecallOK below) - part of the "best" ranking itself,
//--- not just a side note, so blended accuracy alone can never outrank a directionally-useful era
//--- (see the checkpoint/eta-decay comment in Train()'s era-end block for why that matters).
bool m_bestPassedRecall;
bool m_haveOosCheckpoint;
bool m_oosStable;
bool m_objectiveMet;
uint m_syncWaitStartTick; // 0 = not waiting on history sync; else GetTickCount() when the wait began
//--- 3 no-op passes on a fresh start (see InitNeuralNetwork()/ResetWeights()), each its own separately-
//--- scheduled Train() call (not a tight in-process loop), so the broker/terminal's history sync gets
//--- several real, wall-clock-separated chances to finish before the era loop commits to a bar count.
int m_warmupPassesRemaining;
//--- fractal/swing-confirmation/trend-context Buy/Sell label cache: the label at a given now-relative
//--- bar index only depends on price/ATR history, never on model state, so recomputing it every era
//--- (as opposed to once per real bar close) is pure waste. Rebuilt fresh whenever `bars` changes
//--- (see the invalidation check in Train()) rather than incrementally appended, since MQL5 timeseries
//--- indices are relative to "now" and shift by one on every new candle - a full rebuild on change
//--- sidesteps needing datetime-keyed/incremental bookkeeping entirely.
bool m_labelCacheBuy[];
bool m_labelCacheSell[];
bool m_labelCacheHasValue[];
int m_labelCacheBars; // 0 = no cache built yet
datetime m_labelCacheAnchorTime; // m_Time.GetData(0) at last (re)build - 2nd invalidation key
void ComputeLabelForBar(int i, int bars, bool &buy, bool &sell);
void AdvanceZigZagLabelState(int i, int bars);
//--- full per-bar INPUT feature vector cache (everything BufferTempData() computes: ATR-normalized
//--- OHLC, time-of-day encoding, volume delta, AD indicator buffers, ...). Same rationale as the
//--- label cache above - a given now-relative bar index's feature vector only depends on price/
//--- indicator history, never on model state, so recomputing it is pure waste: BufferTempData()
//--- used to rebuild every bar from scratch once per historyBars-wide window it appears in, AND
//--- again on every subsequent era on top of that. Flat array (idx*m_neuronsCount + feature),
//--- not a true 2D array - MQL5 can't dynamically resize an inner dimension. Shares the label
//--- cache's invalidation trigger (see EnsureBarCachesCapacity()) since both are keyed on the exact
//--- same now-relative index frame.
double m_featureCache[];
bool m_featureCacheHasValue[]; // true once idx has been resolved (valid OR invalid)
bool m_featureCacheValid[]; // false: BufferTempDataCompute(idx) returned false (e.g.
// no ATR yet) - cached as a miss so it isn't retried forever
bool BufferTempDataCompute(int idx);
//--- Nearest confirmed (non-repainting) ZigZag pivot at or after fromIdx - see this method's
//--- definition comment and m_useSwingContext's declaration comment for the repainting-embargo
//--- rationale callers must apply to fromIdx before calling this.
bool FindConfirmedZigZagPivot(int fromIdx, int &pivotIdx, double &pivotPrice, bool &pivotIsLow);
bool EnsureBarCachesCapacity(int bars);
//--- Eager label-cache pre-build + true-label tally, run once per fresh start (see
//--- m_warmupPassesRemaining) BEFORE era 0's real training loop begins. Without this, era 0 trains
//--- with m_prevEraTrueBuyCount/Sell/Neutral all still 0 (see that member's declaration comment),
//--- so the class-balanced oversampler falls back to reps=1 for every class on era 0 specifically -
//--- i.e. era 0 gets ZERO correction for label imbalance, unlike every era after it. Pre-scanning the
//--- whole IS window's true labels upfront (reusing the same cache/ComputeLabelForBar() the era loop
//--- itself uses) lets era 0 start with a real oversampling ratio instead of an uncorrected one.
bool m_labelCachePrebuilt; // true once the one-time pre-scan has completed
bool m_labelPrebuildActive; // true while a chunked pre-scan is in progress
bool m_prebuildSeedPending; // true: era 0's era-start reset must NOT stomp the
// prebuild-seeded m_prevEraTrue* counts with the
// still-empty live tally (see Train()'s era-start block)
int m_labelPrebuildBars;
int m_labelPrebuildOosCutoff;
int m_labelPrebuildIndex;
int m_labelPrebuildBuyCount;
int m_labelPrebuildSellCount;
int m_labelPrebuildNeutralCount;
void StartLabelCachePrebuild(void);
void AdvanceLabelCachePrebuild(void);
//--- Evaluation-only continual-learning OOS simulation: once the core model converges, a CLONE of its
//--- weights (never the production Net itself) walks forward through the OOS window bar-by-bar,
//--- scoring each bar with its current weights THEN learning from it - simulating how the model would
//--- adapt in live/forward trading. This must never feed back into Net or the real OOS convergence
//--- metric (dOosForecast/m_oosSamples), so it's tracked in entirely separate members and the clone is
//--- discarded (never Save()'d) once each walk completes.
CNet *m_simOosNet; // NULL when no simulation is active
bool m_simOosRunActive;
int m_simOosCutoff; // oosCutoff snapshot from the run that converged
int m_simOosBarIndex; // resume point, m_simOosCutoff-1 down to 0
double m_simOosForecast; // smoothed accuracy - separate from dOosForecast
int m_simOosSamples;
void StartOosContinualSimulation(int bars, int oosCutoff);
void AdvanceOosSimulationChunk(void);
//--- same resumability problem one level up: TuneIndicatorsAndTrain()'s own trial loop calls
//--- Train() per trial and used to assume each call ran an entire trial to completion synchronously
int m_tuneTrialIndex; // -1 = no multi-trial tuning run in progress
double m_tuneBestOosForecast;
bool m_tuneLastTrialWasWin;
bool m_tuneHaveBestCheckpoint;
datetime m_tuneStartTrainBar;
void FinalizeTrainRun(void);
//--- variables
//--- training control, driven by the control panel (Warrior_EA.mq5); Train()/OnTickHandler
//--- poll these rather than being torn down/rebuilt, so pausing/stopping never loses in-memory state
bool m_trainingPaused; // true: Train() blocks between eras until unpaused
bool m_trainingStopRequested; // true: OnTickHandler stops scheduling new training passes
string m_fileName;
string m_folderPath;
//--- which file Train()'s Net.Save() calls (and this method's own Net.Load()) actually target:
//--- the shared FILE_COMMON production weights normally, or a LOCAL per-agent cache file when
//--- running inside the Strategy Tester/optimizer (see InitNeuralNetwork) so that repeated
//--- optimization passes with an unchanged topology can reuse an already-trained model instead of
//--- re-running every era from scratch, without ever touching the live production .nnw/.cfg.
string m_activeFileName;
bool m_activeFileCommon;
//--- user-settable via Inputs.mqh's TrainingOptimizer (SGD or ADAM), read into this member at
//--- construction. Honored by all three signal types - PAI/CONV via BuildFreshTopology(), and
//--- LSTM via CSignalLSTM::AddCustomLayers - since CNeuronLSTMOCL (AI\Network.mqh) now has an
//--- accelerated SGD+momentum path (LSTM_UpdateWeightsMomentum) alongside its original Adam-only
//--- one, and CNet::CNet's GPU/DirectML init gate no longer restricts LSTM topologies to ADAM.
int m_optimizationAlgo;
int m_historyBars;
int m_outputNeuronsCount;
int m_minNeuronsCount;
int m_initialNeuronsCount;
int m_neuronsCount;
double m_neuronsReduction;
int m_hiddenLayersCount;
int m_studyPeriod;
int m_minTrainYear;
bool m_isInitialized;
int m_stopTrainWR;
int m_fractalPeriods;
//--- "weight" of the (single) market model produced by the network
int m_pattern_0;
//--- functions
virtual bool InitIndicators(CIndicators *indicators);
//--- sets ID/m_id/m_folderPath/m_fileName/m_pattern_count from the subclass constructor
void SetIdentity(string id, string shortId, int patternCount = 1);
//--- hook for neuron-type-specific layers (Conv+Pool, LSTM, ...); default is a plain perceptron (no-op)
virtual bool AddCustomLayers(CArrayObj *topology) { return true; }
//--- hardcoded activation for the common tapering Dense hidden-layer stack built by
//--- BuildFreshTopology() (below AddCustomLayers, above the output layer). PRELU (leaky ReLU,
//--- 0.01 slope) is the default: it doesn't saturate/vanish the way TANH does across a deep
//--- taper, and is what the Conv layer itself already uses (see CSignalCONV::AddCustomLayers).
//--- CSignalLSTM overrides this to TANH instead - these Dense layers sit directly on top of the
//--- LSTM layer's own TANH-bounded ([-1,1]) output, so keeping them bounded too avoids feeding an
//--- unbounded activation straight off a bounded recurrent output, which is untested territory
//--- here and not what the user asked for ("Tanh for LSTM gates").
virtual ENUM_ACTIVATION HiddenLayerActivation(void) { return PRELU; }
//--- common network bootstrap: indicators, topology build/load, training-file bookkeeping
bool InitNeuralNetwork(CIndicators *indicators);
void DrawObject(datetime time, double signal, double high, double low);
void DeleteObject(datetime time);
void PurgeChart(void);
ENUM_SIGNAL DoubleToSignal(double value);
//--- Shared status-label formatting for all three of Train()'s era passes (pass 1 sequential scan/
//--- display, pass 2 shuffled backProp, pass 3 post-training OOS scoring) - see m_isTrainQueue's and
//--- m_isPass2Active's declaration comments for why the era loop is now three passes instead of one.
//--- Extracted so the panel keeps updating every bar throughout ALL three passes instead of freezing
//--- during pass 2/3 the way it did when this was pass 1-only inline code - a stalled-looking panel
//--- during a still-running era was reported as looking "stuck". Throttled internally (see
//--- m_lastStatusLabelUpdateTick) - SetStatusLabel() (System\StatusLabel.mqh) does real text-layout
//--- work (TextGetSize/WrapLineInto per line) AND an unconditional ChartRedraw(0) every call, which
//--- gets slower as more chart objects accumulate over a long backtest. Calling it unconditionally
//--- once per bar was already the existing (pass-1-only) behavior; doing that again independently in
//--- pass 2 AND pass 3 roughly TRIPLED total ChartRedraw() calls per era and was the actual cause of
//--- a training run observed taking 1.5+ hours without completing era 0 - not the shuffle itself.
//--- forceRefresh=true bypasses the throttle below - used exactly once per era, right after the
//--- era-end block (Train()) finalizes m_eraCount/dOosForecast, so the panel's "Era %d"/"OOS Acc"
//--- fields update in the SAME moment the console's "training in progress - era N" line does.
//--- Without it, the panel's last real redraw during a normal (throttled) bar-scan call happened
//--- DURING pass 3, before m_eraCount++ - so it kept showing era N-1's number (with what was, by
//--- then, already era N's near-final accuracy) for this era's entire duration, only catching up to
//--- the correct era number once the NEXT era's own bar-scan calls started - a full one-era-behind
//--- display lag relative to the console log, reported in practice.
void UpdateTrainingStatusLabel(const string &progressLine, double neuron0, double neuron1, double neuron2, double signalValue, bool forceRefresh = false);
//--- Per-instance (NOT a function-local static - see CExpertSignalCustom::Direction()'s declaration
//--- comment for why that distinction matters for a method shared across PAI/CONV/LSTM instances)
//--- wall-clock throttle gate for UpdateTrainingStatusLabel()'s ChartRedraw().
uint m_lastStatusLabelUpdateTick;
//--- Last values passed to UpdateTrainingStatusLabel() - cached (updated on EVERY call, throttled
//--- or not) so the forced era-end refresh above has something real to redraw with instead of a
//--- stale/zeroed placeholder, since no "current bar" exists once an era's own three passes are done.
double m_lastDisplayNeuron0, m_lastDisplayNeuron1, m_lastDisplayNeuron2, m_lastDisplaySignal;
//--- turns the classification output layer's 3 values (TempData[0..2], SIGMOID activation - see
//--- BuildFreshTopology() - each already in [0,1]) into a softmax probability distribution in
//--- place, and returns the signed dPrevSignal convention (+P(buy), -P(sell), 0.0 exactly for
//--- neutral) used by both Train()'s live-forecast branch and RefreshLatestSignal(). Softmax is
//--- monotonic per-element so it can't change which of the 3 wins - it only exists to turn the 3
//--- independently-trained values into a normalized confidence. Max-subtracted before exp() for
//--- numerical stability regardless (safe since softmax(x) == softmax(x - max(x))).
double ApplyClassificationSoftmax(void);
bool ResizeBuffers(int barIndex);
bool RefreshData();
bool BufferTempData(int idx);
//--- shared by OnTickHandler() and the timer-driven PollTraining() - see definition
void ScheduleTrainingIfNeeded(void);
void Train(datetime StartTrainBar = 0);
//--- outer loop around Train(): when AutoTuneIndicators is on, tries randomized AD indicator
//--- input variations across m_indicatorTuneTrials calls to Train(), keeping the best-OOS one
void TuneIndicatorsAndTrain(datetime StartTrainBar = 0);
//--- recomputes dPrevSignal/chart arrow for the most recent bar; used after restoring a
//--- checkpointed model at the end of Train() so the live signal matches the deployed weights
void RefreshLatestSignal();
//--- inference-only "new bar" handler used once m_trainingComplete is true - see
//--- ScheduleTrainingIfNeeded()'s declaration comment for why this must NOT call Net.backProp()
void RefreshConvergedSignal(void);
//--- lazily bootstraps m_shadowNet if it's still NULL: tries loading a persisted shadow file
//--- first (continuity across EA restarts), falling back to cloning Net's current weights (via
//--- the same Save()/Load() pattern StartOosContinualSimulation() uses for m_simOosNet) if no
//--- compatible shadow file exists yet. No-op if m_shadowNet is already valid. See m_shadowNet's
//--- declaration comment for the full EMA shadow-weight deployment rationale.
void EnsureShadowNet(void);
//--- persists m_shadowNet alongside every Net.Save() call, using the same run metadata (error/
//--- undefine/forecast/era/trainingComplete/indicator params) the caller already computed for
//--- Net.Save() itself - see m_shadowNet's declaration comment. No-op if the shadow isn't
//--- bootstrapped yet.
void SaveShadowNet(const double &indicatorParams[]);
//--- method of initialization of the indicators
bool InitOpen(CIndicators *indicators);
bool InitClose(CIndicators *indicators);
bool InitHigh(CIndicators *indicators);
bool InitLow(CIndicators *indicators);
bool InitVolumes(CIndicators *indicators);
bool InitTime(CIndicators *indicators);
//--- addToCollection=false is used by ReInitADIndicators() to rebuild an already-collected
//--- handle's params without re-adding the (same) pointer into indicators a second time
bool InitADCumulativeDelta(CIndicators *indicators, bool addToCollection = true);
bool InitADShorteningOfThrust(CIndicators *indicators, bool addToCollection = true);
bool InitADWyckoffEventStream(CIndicators *indicators, bool addToCollection = true);
bool InitADWyckoffFailedStructure(CIndicators *indicators, bool addToCollection = true);
bool InitADWyckoffSignificantBarInversion(CIndicators *indicators, bool addToCollection = true);
bool InitADZigZag(CIndicators *indicators, bool addToCollection = true);
//--- common=false targets a LOCAL (non-shared) file - used by the tester/optimizer per-agent
//--- weight cache so cross-pass reuse never touches the production FILE_COMMON config/weights.
bool SaveTopologyConfiguration(string fileName, int initialNeuronsCount, int hiddenLayersCount, double neuronsReduction, int minNeuronsCount, int optimizationAlgo, int historyBars, int outputNeuronsCount, int neuronsCount, int studyPeriod, int minTrainYear, bool isInitialized, int stopTrainWR, int fractalPeriods, bool common = true);
bool LoadAndCompareTopologyConfiguration(string fileName, int initialNeuronsCount, int hiddenLayersCount, double neuronsReduction, int minNeuronsCount, int optimizationAlgo, int historyBars, int outputNeuronsCount, int neuronsCount, int studyPeriod, int minTrainYear, bool isInitialized, int stopTrainWR, int fractalPeriods, bool common = true);
//--- input data
bool m_useVolumes;
bool m_useTime;
bool m_useATR;
//--- Normalized ZigZag swing-context features (direction/magnitude/age of the last CONFIRMED swing -
//--- see BufferTempDataCompute()'s m_useSwingContext block for the exact 5 values). Reads the same
//--- m_ADZigZag the training labels already come from (see that member's declaration comment) rather
//--- than a separate indicator instance - NOT gated behind AutoTuneIndicators/ReInitADIndicators like
//--- the AD* indicators below, since m_ADZigZag itself is deliberately never tuned (same reason as
//--- the label side: tuning the ground truth alongside the model scored against it would let a trial
//--- cherry-pick an easier target). Critical correctness constraint, not just a style choice: ZigZag's
//--- most recent 1-3 legs repaint (see m_swingConfirmationBars' declaration comment), so every read of
//--- m_ADZigZag for THIS feature - exactly like the label side - must only trust a pivot that is
//--- already at least m_swingConfirmationBars bars old relative to the bar the feature is being
//--- computed for. Skipping that embargo would leak future information a live bar couldn't actually
//--- have had yet - silent lookahead bias inflating backtest/training performance without being real.
bool m_useSwingContext;
//--- see System\NewsRelevance.mqh's declaration comment for what this feature actually encodes
//--- (event proximity + impact, not actual-vs-forecast deviation) and why the forward-looking half
//--- of it isn't lookahead bias.
bool m_useNews;
int m_newsFeatureWindowMinutes;
bool m_useADCumulativeDelta;
bool m_useADShorteningOfThrust;
bool m_useADWyckoffEventStream;
bool m_useADWyckoffFailedStructure;
bool m_useADWyckoffSignificantBarInversion;
public:
CExpertSignalAIBase(void);
~CExpertSignalAIBase(void);
//--- "voting" that price will grow/fall, common to every AI signal (single market model)
virtual int LongCondition(void);
virtual int ShortCondition(void);
// |dPrevSignal| is already a 0..1 confidence for classification output (softmax
// probability of the winning class) and typically bounded for regression output
// (tanh-activated network); OpenParams() clamps regardless. Scaled by m_confidenceCalScale
// (classification head only - 1.0/no-op for regression) so callers get an empirically
// calibrated magnitude instead of the raw, uncalibrated softmax value - see
// m_confidenceCalScale's declaration comment.
double CalibratedConfidenceMagnitude(void) const
{
double mag = MathAbs(dPrevSignal);
if(m_outputNeuronsCount == 3)
mag = MathMin(1.0, mag * m_confidenceCalScale);
return mag;
}
virtual double AIConfidence(void) override { return CalibratedConfidenceMagnitude(); }
// Signed for direction-aware use (AI-driven early exit): sign matches dPrevSignal's
// convention (+ buy, - sell, 0 neutral/no signal yet). dPrevSignal == -2 is the
// "not yet studied" sentinel, not a real sell signal - treat it as no confidence.
virtual double SignedAIConfidence(void) override
{
if(dPrevSignal == -2)
return 0.0;
double sign = (dPrevSignal > 0.0) ? 1.0 : (dPrevSignal < 0.0) ? -1.0 : 0.0;
return sign * CalibratedConfidenceMagnitude();
}
//--- event handlers, common to every AI signal
virtual void OnTickHandler(void);
//--- drives the same training-scheduling check as OnTickHandler(), but callable from a timer so
//--- it isn't dependent on ticks (which don't arrive while the market is closed)
void PollTraining(void);
virtual void OnChartEventHandler(const int id,
const long &lparam,
const double &dparam,
const string &sparam);
//--- methods of adjusting "weight" of the market model
void Pattern_0(int value) { m_pattern_0 = value; }
virtual void ApplyPatternWeight(int patternNumber, int weight);
//--- methods of setting adjustable parameters
void InitialNeuronsCount(int value) { m_initialNeuronsCount = value; }
void OutputNeuronsCount(int value) { m_outputNeuronsCount = value; }
void MinNeuronsCount(int value) { m_minNeuronsCount = value; }
void HiddenLayersCount(int value) { m_hiddenLayersCount = value; }
void NeuronsReduction(int value) { m_neuronsReduction = value; }
void StopTrainWR(int value) { m_stopTrainWR = value; }
void MinDirectionalRecall(int value) { m_minDirectionalRecallPct = value; }
void ClassSampleWeight(double value) { m_maxClassSampleWeight = value; }
void FocalLossGamma(double value) { m_focalGamma = value; }
void SwingConfirmationBars(int value) { m_swingConfirmationBars = value; }
void MaxErasPerRun(int value) { m_maxErasPerRun = value; }
void TrainRetryCooldownSeconds(int value) { m_trainRetryCooldownSeconds = value; }
void OOSSplit(int value) { m_oosSplitPct = value; }
void HistoryBars(int value) { m_historyBars = value; }
void StudyPeriod(int value) { m_studyPeriod = value; }
void MinTrainYear(int value) { m_minTrainYear = value; }
void UseVolumes(bool value) { m_useVolumes = value; }
void UseTime(bool value) { m_useTime = value; }
void UseATR(bool value) { m_useATR = value; }
void UseSwingContext(bool value) { m_useSwingContext = value; }
void UseNews(bool value) { m_useNews = value; }
void NewsFeatureWindowMinutes(int value) { m_newsFeatureWindowMinutes = value; }
void UseADCumulativeDelta(bool value) { m_useADCumulativeDelta = value; }
void UseADShorteningOfThrust(bool value) { m_useADShorteningOfThrust = value; }
void UseADWyckoffEventStream(bool value) { m_useADWyckoffEventStream = value; }
void UseADWyckoffFailedStructure(bool value) { m_useADWyckoffFailedStructure = value; }
void UseADWyckoffSignificantBarInversion(bool value) { m_useADWyckoffSignificantBarInversion = value; }
void AutoTuneIndicators(bool value) { m_autoTuneIndicators = value; }
void IndicatorTuneTrials(int value) { m_indicatorTuneTrials = value; }
//--- control-panel API (Warrior_EA.mq5): current-config-only training/weights control.
//--- "current config" == this signal instance's own m_fileName (symbol+period+id+topology),
//--- never touches another signal type's or another symbol/timeframe's saved files.
void PauseTraining(void) { m_trainingPaused = true; PrintVerbose(ID + ": training paused by user (era " + IntegerToString(m_eraCount) + ")"); }
void ResumeTraining(void) { m_trainingPaused = false; PrintVerbose(ID + ": training resumed by user (era " + IntegerToString(m_eraCount) + ")"); }
bool IsTrainingPaused(void) const { return m_trainingPaused; }
bool IsTrainingStopped(void) const { return m_trainingStopRequested; }
long EraCount(void) const { return m_eraCount; }
bool TrainingComplete(void) const { return m_trainingComplete; }
void StopTraining(void)
{
m_trainingStopRequested = true;
m_trainingPaused = false;
//--- ScheduleTrainingIfNeeded() refuses to schedule another "New Bar" event while
//--- m_trainingStopRequested is set, so a run interrupted mid-chunk would otherwise never get
//--- called again to finalize (restore the best checkpoint, persist state) - do it synchronously
//--- here instead. Safe to block briefly: this is just a checkpoint file restore, not the
//--- multi-minute bar loop.
if(m_trainRunActive)
FinalizeTrainRun();
Print(ID + ": training stopped by user (era " + IntegerToString(m_eraCount) + ", weights as of last completed era retained)");
}
void StartTraining(void)
{
if(!m_trainingStopRequested && !m_trainingPaused)
return;
m_trainingStopRequested = false;
m_trainingPaused = false;
if(!bEventStudy)
bEventStudy = EventChartCustom(ChartID(), 1, (long)dtStudied, 0, "Resume");
Print(ID + ": training (re)started by user (era " + IntegerToString(m_eraCount) + ")");
}
//--- forces a save of the network's current in-memory weights/state regardless of era-completion
//--- state; called from OnDeinit() so shutdown/chart-removal never loses more than the current tick
//--- of learning, and a subsequent restart's Train() resumes from m_eraCount rather than the last
//--- fully-completed era only.
bool PersistOnShutdown(void)
{
if(CheckPointer(Net) == POINTER_INVALID || !m_isInitialized)
return false;
double currentIndicatorParams[];
m_indicatorTuner.Flatten(currentIndicatorParams);
bool ok = Net.Save(m_activeFileName + ".nnw", dError, dUndefine, dForecast, dtStudied, m_activeFileCommon, m_eraCount, m_trainingComplete, currentIndicatorParams);
SaveShadowNet(currentIndicatorParams);
if(!ok)
Print(ID + ": ERROR - failed to persist weights on shutdown for " + m_activeFileName + ", error " + IntegerToString(GetLastError()));
else
PrintVerbose(ID + ": weights persisted on shutdown (era " + IntegerToString(m_eraCount) + ", trainingComplete=" + (string)m_trainingComplete + ")");
return ok;
}
//--- explicit manual save, identical persistence to PersistOnShutdown() but user-triggered from the panel
bool SaveWeightsNow(void) { return PersistOnShutdown(); }
//--- reloads this signal's current-config weights file from disk, discarding any unsaved in-memory
//--- training progress since the last successful save
bool LoadWeightsNow(void)
{
if(CheckPointer(Net) == POINTER_INVALID)
return false;
double loadedIndicatorParams[];
bool netLoaded = Net.Load(m_activeFileName + ".nnw", dError, dUndefine, dForecast, dtStudied, m_activeFileCommon, m_eraCount, m_trainingComplete, loadedIndicatorParams);
if(!netLoaded)
{
Print(ID + ": ERROR - failed to load weights from " + m_activeFileName + ".nnw, error " + IntegerToString(GetLastError()));
return false;
}
if(ArraySize(loadedIndicatorParams) == AD_TUNE_PARAM_COUNT)
{
m_indicatorTuner.Unflatten(loadedIndicatorParams);
if(m_indicatorsPtr != NULL)
ReInitADIndicators(m_indicatorsPtr);
}
//--- this just swapped dtStudied/m_eraCount/Net's weights out from under whatever a chunked
//--- Train() run (see its declaration comment) had cached for the era/trial it was mid-way
//--- through - discard that resumable state so the next Train() call starts a fresh era
//--- against the just-loaded dtStudied instead of resuming bar-loop bookkeeping computed
//--- against a now-stale one.
m_trainRunActive = false;
m_eraResumePending = false;
m_haveOosCheckpoint = false;
m_oosWindow.Clear();
m_tuneTrialIndex = -1;
RefreshLatestSignal();
Print(ID + ": weights reloaded from disk (era " + IntegerToString(m_eraCount) + ", trainingComplete=" + (string)m_trainingComplete + ")");
return true;
}
//--- deletes this signal's current-config saved files only (weights, topology config, in-progress
//--- checkpoint) and rebuilds a fresh untrained topology in memory so training restarts from era 0.
//--- Never touches another signal type's or another symbol/timeframe's files - m_fileName already
//--- embeds symbol+period+id+output-count+opt-algo.
bool ResetWeights(void)
{
bool stopped = m_trainingStopRequested;
m_trainingStopRequested = true; // hold off any in-flight Train() scheduling while we reset
//--- targets whichever file this run is actually training against (see InitNeuralNetwork): the
//--- shared production weights normally, or the local tester/optimizer cache during a backtest -
//--- so resetting from the panel during a visual-mode backtest can never wipe the live model.
int flags = m_activeFileCommon ? FILE_COMMON : 0;
string nnw = m_activeFileName + ".nnw";
string cfg = m_activeFileName + ".cfg";
string ckpt = m_activeFileName + "_ckpt.tmp";
ResetLastError();
if(FileIsExist(nnw, flags) && !FileDelete(nnw, flags))
Print(ID + ": ERROR - failed to delete " + nnw + ", error " + IntegerToString(GetLastError()));
ResetLastError();
if(FileIsExist(cfg, flags) && !FileDelete(cfg, flags))
Print(ID + ": ERROR - failed to delete " + cfg + ", error " + IntegerToString(GetLastError()));
ResetLastError();
if(FileIsExist(ckpt) && !FileDelete(ckpt))
Print(ID + ": ERROR - failed to delete " + ckpt + ", error " + IntegerToString(GetLastError()));
m_eraCount = 0;
m_trainingComplete = false;
dtStudied = 0;
dError = -1;
dUndefine = 0;
dForecast = 0;
dPrevSignal = 0;
m_lastNonNeutralSignal = Neutral;
dOosError = -1;
dOosForecast = 0;
m_oosSamples = 0;
//--- discard any in-progress chunked run/tuning state - it references buffers/checkpoints from
//--- before this reset and must never be resumed into the freshly rebuilt topology below
m_trainRunActive = false;
m_eraResumePending = false;
m_haveOosCheckpoint = false;
m_oosWindow.Clear();
m_syncWaitStartTick = 0;
m_tuneTrialIndex = -1;
//--- an explicit reset restarts from era 0 against a fresh topology - re-verify history sync and
//--- rebuild the label cache too, and drop any in-flight continual-learning OOS simulation, since
//--- both would otherwise reference bars/weights from before this reset.
m_warmupPassesRemaining = 3;
m_labelCacheBars = 0;
m_labelCacheAnchorTime = 0;
m_labelCachePrebuilt = false;
m_labelPrebuildActive = false;
m_prebuildSeedPending = false;
if(m_simOosRunActive)
{
delete m_simOosNet;
m_simOosNet = NULL;
m_simOosRunActive = false;
}
bool rebuilt = BuildFreshTopology();
if(!rebuilt)
Print(ID + ": ERROR - failed to rebuild fresh topology after weights reset");
else
{
SaveTopologyConfiguration(m_activeFileName, m_initialNeuronsCount, m_hiddenLayersCount, m_neuronsReduction, m_minNeuronsCount, m_optimizationAlgo, m_historyBars, m_outputNeuronsCount, m_neuronsCount, m_studyPeriod, m_minTrainYear, m_isInitialized, m_stopTrainWR, m_fractalPeriods, m_activeFileCommon);
Print(ID + ": weights reset - training will restart from era 0 (current config only: " + m_activeFileName + ")");
}
m_trainingStopRequested = stopped;
if(!stopped && !bEventStudy)
bEventStudy = EventChartCustom(ChartID(), 1, 0, 0, "Reset");
return rebuilt;
}
};
//+------------------------------------------------------------------+
//| Constructor |
//+------------------------------------------------------------------+
CExpertSignalAIBase::CExpertSignalAIBase(void) :
ID("NULL"),
m_neuronsCount(0),
m_studyPeriod(10),
m_minTrainYear(1970),
m_optimizationAlgo(TrainingOptimizer), // see the member declaration comment
m_initialNeuronsCount(1000),
m_outputNeuronsCount(3),
m_minNeuronsCount(20),
m_neuronsReduction(0.5),
m_hiddenLayersCount(4),
m_historyBars(14),
m_stopTrainWR(70),
m_fractalPeriods(5),
m_pattern_0(10),
m_useVolumes(true),
m_useTime(true),
m_useATR(true),
m_useSwingContext(false),
m_useNews(false),
m_newsFeatureWindowMinutes(60),
m_useADCumulativeDelta(false),
m_useADShorteningOfThrust(false),
m_useADWyckoffEventStream(false),
m_useADWyckoffFailedStructure(false),
m_useADWyckoffSignificantBarInversion(false),
m_autoTuneIndicators(false),
m_indicatorTuneTrials(8),
m_indicatorsPtr(NULL),
Net(NULL),
m_shadowNet(NULL),
TempData(NULL),
dError(-1),
dUndefine(0),
dForecast(0),
dPrevSignal(0),
m_lastNonNeutralSignal(Neutral),
dtStudied(0),
m_eraCount(0),
m_trainingComplete(false),
bEventStudy(false),
m_oosSplitPct(30),
dOosError(-1),
dOosForecast(0),
m_oosSamples(0),
m_countBuySignals(0),
m_countSellSignals(0),
m_countNeutralSignals(0),
m_trueBuyCount(0),
m_trueSellCount(0),
m_trueNeutralCount(0),
m_prevEraTrueBuyCount(0),
m_prevEraTrueSellCount(0),
m_prevEraTrueNeutralCount(0),
m_oosBuyHits(0),
m_oosBuyTotal(0),
m_oosSellHits(0),
m_oosSellTotal(0),
m_oosNeutralHits(0),
m_oosNeutralTotal(0),
m_oosBuyPredicted(0),
m_oosBuyPredictedHits(0),
m_oosSellPredicted(0),
m_oosSellPredictedHits(0),
m_oosNeutralPredicted(0),
m_oosNeutralPredictedHits(0),
m_oosConfidenceSum(0),
m_confidenceCalScale(1.0),
m_minDirectionalRecallPct(40),
m_maxClassSampleWeight(1.5),
m_focalGamma(2.0),
m_swingConfirmationBars(100),
m_maxErasPerRun(300),
m_trainRetryCooldownSeconds(60),
m_trainCooldownUntil(0),
m_trainRunActive(false),
m_eraResumePending(false),
m_resumeBars(0),
m_resumeTotalIter(0),
m_resumeOosCutoff(0),
m_resumeBarIndex(0),
m_resumeAddLoop(false),
m_isTrainQueueCount(0),
m_isTrainCursor(0),
m_isPass2Active(false),
m_isPass2Done(false),
m_isPass3Active(false),
m_oosScoreIndex(0),
m_oosScoreStartIndex(0),
m_lastStatusLabelUpdateTick(0),
m_lastDisplayNeuron0(0),
m_lastDisplayNeuron1(0),
m_lastDisplayNeuron2(0),
m_lastDisplaySignal(0),
m_lastBarTime(0),
m_modelEta(InitialEtaForOptimizer()),
m_etaCeiling(InitialEtaForOptimizer()),
m_erasSinceCooldown(0),
m_bestOosForecast(-1),
m_bestPassedRecall(false),
m_haveOosCheckpoint(false),
m_oosStable(false),
m_objectiveMet(false),
m_syncWaitStartTick(0),
m_warmupPassesRemaining(0),
m_labelCacheBars(0),
m_labelCacheAnchorTime(0),
m_labelCachePrebuilt(false),
m_labelPrebuildActive(false),
m_prebuildSeedPending(false),
m_labelPrebuildBars(0),
m_labelPrebuildOosCutoff(0),
m_labelPrebuildIndex(-1),
m_labelPrebuildBuyCount(0),
m_labelPrebuildSellCount(0),
m_labelPrebuildNeutralCount(0),
m_simOosNet(NULL),
m_simOosRunActive(false),
m_simOosCutoff(0),
m_simOosBarIndex(-1),
m_simOosForecast(0),
m_simOosSamples(0),
m_tuneTrialIndex(-1),
m_tuneBestOosForecast(-1),
m_tuneLastTrialWasWin(true),
m_tuneHaveBestCheckpoint(false),
m_tuneStartTrainBar(0),
m_trainingPaused(false),
m_trainingStopRequested(false),
m_activeFileCommon(true),
m_isInitialized(false)
{
//--- indicator tuning defaults live in CADIndicatorTuner's own constructor (Expert\ADIndicatorTuner.mqh),
//--- which runs automatically for the m_indicatorTuner member above.
}
//+------------------------------------------------------------------+
//| Destructor |
//+------------------------------------------------------------------+
CExpertSignalAIBase::~CExpertSignalAIBase(void)
{
//--- deliberately NOT calling PersistOnShutdown() here: OnDeinit() (Warrior_EA.mq5) already calls
//--- it explicitly for every signal, one call stack frame shallower, BEFORE Expert.Deinit() tears
//--- these objects down. Doing it again here nested inside that same teardown cascade doubled the
//--- stack depth of an already-deep recursive save (layers -> neurons -> connections) right at the
//--- point in MT5's lifecycle (EA recompile while attached) that has the least stack headroom, and
//--- reliably crashed the terminal with a stack overflow. Keep this destructor cheap.
if(CheckPointer(Net) != POINTER_INVALID)
delete Net;
if(CheckPointer(m_shadowNet) != POINTER_INVALID)
delete m_shadowNet;
if(CheckPointer(TempData) != POINTER_INVALID)
delete TempData;
if(CheckPointer(m_simOosNet) != POINTER_INVALID)
delete m_simOosNet;
PurgeChart();
}
//+------------------------------------------------------------------+
//| Sets the file/id identity a subclass constructor would otherwise |
//| repeat verbatim (ID, m_id, m_folderPath, m_fileName, pattern count)|
//+------------------------------------------------------------------+
void CExpertSignalAIBase::SetIdentity(string id, string shortId, int patternCount = 1)
{
ID = id;
m_id = shortId;
m_folderPath = eaName + "\\" + "Neural Networks" + "\\" + "State" + "\\" + m_id + "\\";
m_fileName = m_folderPath + _Symbol + "_" + IntegerToString(_Period);
m_pattern_count = patternCount;
}
//+------------------------------------------------------------------+
//| "Voting" that price will grow. |
//+------------------------------------------------------------------+
int CExpertSignalAIBase::LongCondition(void)
{
int result = 0;
//--- alternation gate - see m_lastNonNeutralSignal's declaration comment. Suppressed until a Sell
//--- has been seen since the last Buy we fired.
if(DoubleToSignal(dPrevSignal) == Buy && m_lastNonNeutralSignal != Buy)
{
result = m_pattern_0;
m_active_pattern = "Pattern_0";
m_active_direction = "Buy";
m_lastNonNeutralSignal = Buy;
}
return(result);
}
//+------------------------------------------------------------------+
//| "Voting" that price will fall. |
//+------------------------------------------------------------------+
int CExpertSignalAIBase::ShortCondition(void)
{
int result = 0;
//--- alternation gate - see m_lastNonNeutralSignal's declaration comment. Suppressed until a Buy
//--- has been seen since the last Sell we fired.
if(DoubleToSignal(dPrevSignal) == Sell && m_lastNonNeutralSignal != Sell)
{
result = m_pattern_0;
m_active_pattern = "Pattern_0";
m_active_direction = "Sell";
m_lastNonNeutralSignal = Sell;
}
return result;
}
//+------------------------------------------------------------------+
//| Set the specified pattern's weight to the specified value |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::ApplyPatternWeight(int patternNumber, int weight)
{
switch(patternNumber)
{
case 0:
Pattern_0(weight);
break;
default:
break;
}
}
//+------------------------------------------------------------------+
//| OnTick function |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::OnTickHandler(void)
{
ScheduleTrainingIfNeeded();
}
//+------------------------------------------------------------------+
//| Schedules the next training pass (if one is due) and refreshes |
//| the per-tick status label. Factored out of OnTickHandler() so |
//| Warrior_EA.mq5's always-on timer (see PollTraining()) can drive |
//| this on a fixed wall-clock schedule too - training must not stall |
//| just because the market is closed and no ticks are arriving. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::ScheduleTrainingIfNeeded(void)
{
//--- stopped: no new training passes get scheduled at all (StartTraining() re-arms this).
//--- paused: still schedule so bEventStudy/dtStudied bookkeeping stays current, but Train() itself
//--- blocks at the next era boundary until resumed - keeps in-memory state coherent either way.
//--- cooldown: Train() just hit its per-call era safety cap without converging - back off for a
//--- while so the normal per-tick info line below gets a chance to actually show, instead of every
//--- tick immediately re-triggering another long, status-label-dominating training attempt.
//--- complete: training already converged - a plain new bar must NOT re-enter Train()'s full era
//--- loop, which would otherwise reset the best-checkpoint/eta-decay tracking and run real
//--- Net.backProp() passes again, forever, once per bar, on an already-converged model (see
//--- RefreshConvergedSignal()'s declaration comment). Just keep the live signal current instead.
bool inCooldown = (m_trainCooldownUntil > 0 && TimeCurrent() < m_trainCooldownUntil);
datetime lastBarDate = (datetime)SeriesInfoInteger(m_symbol.Name(), m_period, SERIES_LASTBAR_DATE);
// A failed lookup (0) must not silently read as "dtStudied is already caught up, nothing pending" -
// that would freeze this function into never re-triggering training/signal refresh again until some
// other path happens to bump dtStudied. Treat a failed lookup as pending instead (same >0-guard
// philosophy as the SERIES_FIRSTDATE lookup elsewhere in this class) so a transient history-sync
// hiccup costs one extra harmless check, not a silent stall.
bool newBarPending = (dPrevSignal == -2 || lastBarDate <= 0 || dtStudied < lastBarDate);
if(m_trainingComplete && !m_trainingStopRequested && !m_trainRunActive)
{
if(newBarPending)
RefreshConvergedSignal();
}
else
if(!m_trainingStopRequested && !inCooldown && !bEventStudy && newBarPending)
bEventStudy = EventChartCustom(ChartID(), 1, (long)MathMax(0, MathMin(iTime(m_symbol.Name(), PERIOD_CURRENT, (int)(100 * Net.recentAverageSmoothingFactor * (dForecast >= m_stopTrainWR ? 1 : 10))), dtStudied)), 0, "New Bar");
//--- Train() (see its declaration comment) now yields every ~TRAIN_TIME_BUDGET_MS instead of
//--- blocking for a whole era, so while a run is active this per-tick line would otherwise
//--- overwrite Train()'s own full-detail status label on every single tick between chunks -
//--- flickering between the two instead of showing one steady picture. Only write this terse
//--- summary when nothing else is actively updating the status label (idle/stopped/paused/cooldown).
if(!m_trainRunActive)
{
string trainingState = m_trainingStopRequested ? "STOPPED" : (m_trainingPaused ? "PAUSED" : (inCooldown ? "COOLDOWN (hit era cap, retrying soon)" : (m_trainingComplete ? "COMPLETE" : "IN PROGRESS")));
//--- same "Forecast: <signal> -> <value>" line the active training loop's status label ends on
//--- (see the classLine-terminated StringFormat below), instead of a raw bEventStudy/dPrevSignal/
//--- dtStudied debug dump - this is what stays on screen once training stops/pauses/completes.
SetStatusLabel(StringFormat(
ID + " : Era %d -> Training %s\n" +
"Forecast: %s -> %.2f",
m_eraCount, trainingState,
EnumToString(DoubleToSignal(dPrevSignal)), dPrevSignal));
}
}
//+------------------------------------------------------------------+
//| Timer-driven equivalent of OnTickHandler()'s scheduling, called |
//| from Warrior_EA.mq5's always-on OnTimer() so training keeps |
//| progressing purely on wall-clock time - no dependency on ticks, |
//| which simply don't arrive while the market is closed. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::PollTraining(void)
{
if(m_isInitialized)
ScheduleTrainingIfNeeded();
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::OnChartEventHandler(const int id,
const long &lparam,
const double &dparam,
const string &sparam)
{
if(id == 1001)
{
TuneIndicatorsAndTrain(lparam);
bEventStudy = false;
OnTickHandler();
}
}
//+------------------------------------------------------------------+
//| Common network bootstrap shared by every AI signal: sets up |
//| indicators, then loads a saved network or builds a fresh one |
//| whose only per-signal-type difference is AddCustomLayers(). |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::InitNeuralNetwork(CIndicators *indicators)
{
if(m_isInitialized)
return true;
if(indicators == NULL)
return false;
m_indicatorsPtr = indicators;
if(!CExpertSignalCustom::InitIndicators(indicators))
return false;
if(!CExpertSignalAIBase::InitIndicators(indicators))
return false;
Net = new CNet(NULL);
if(CheckPointer(Net) == POINTER_INVALID)
return false;
m_fileName += "_" + DoubleToString(MathRound(m_outputNeuronsCount)) + "_" + DoubleToString(MathRound(m_optimizationAlgo));
//--- Strategy Tester / optimizer: target a LOCAL (agent-sandboxed, non-FILE_COMMON) cache file
//--- instead of the shared production weights, so genetic/complete optimization passes on this
//--- same agent can reuse an already-trained model whenever the topology-relevant inputs
//--- (neuron counts, layers, history bars, output count, opt algo, study period, ...) are
//--- unchanged from a previous pass, instead of re-running every training era from scratch each
//--- pass. The live/manual-chart production .nnw/.cfg under FILE_COMMON are never touched by
//--- this path, so a backtest can never corrupt or overwrite the deployed live model.
bool inTesterOrOpt = MQLInfoInteger(MQL_TESTER) || MQLInfoInteger(MQL_OPTIMIZATION) || MQLInfoInteger(MQL_FORWARD);
m_activeFileName = inTesterOrOpt ? (m_fileName + "_optcache") : m_fileName;
m_activeFileCommon = !inTesterOrOpt;
if(trainingMode && FileIsExist(m_activeFileName + ".nnw", m_activeFileCommon ? FILE_COMMON : 0))
{
int attempts = 0;
bool deleted = false;
while(attempts < 5 && !deleted)
{
if(FileDelete(m_activeFileName + ".nnw", m_activeFileCommon ? FILE_COMMON : 0) && FileDelete(m_activeFileName + ".cfg", m_activeFileCommon ? FILE_COMMON : 0))
{
deleted = true;
break;
}
if(GetLastError() != 5002) // If the error is not because the file does not exist
Print(__FUNCTION__ + ": Retry " + IntegerToString(attempts + 1) + " failed to delete file: " + m_activeFileName);
else
ResetLastError();
Sleep(1000);
attempts++;
}
if(!deleted)
{
if(GetLastError() != 5002)
Print(__FUNCTION__ + ": Failed to delete file after retries: " + m_activeFileName + ".nnw");
return false;
}
// fall through - the save file is gone, so the load below naturally
// misses and a fresh topology gets built instead of leaving Net empty
}
if(!LoadAndCompareTopologyConfiguration(m_activeFileName, m_initialNeuronsCount, m_hiddenLayersCount, m_neuronsReduction, m_minNeuronsCount, m_optimizationAlgo, m_historyBars, m_outputNeuronsCount, m_neuronsCount, m_studyPeriod, m_minTrainYear, m_isInitialized, m_stopTrainWR, m_fractalPeriods, m_activeFileCommon))
{
// Topology/input params diverged from what produced the saved .nnw (or no .cfg exists yet;
// for inTesterOrOpt this is also the normal "first pass on this agent" case). The stale .cfg
// was already deleted on a mismatch, but the .nnw weights themselves are shaped for the OLD
// topology - loading them into a network built to the NEW shape would corrupt state or crash.
// Drop the incompatible weights/checkpoint too so the Net.Load() below cleanly misses and
// BuildFreshTopology() takes over (i.e. this pass pays the training cost once, and the result
// gets cached below for the NEXT pass to reuse, same as a live topology change would).
if(FileIsExist(m_activeFileName + ".nnw", m_activeFileCommon ? FILE_COMMON : 0))
{
Print(__FUNCTION__ + ": " + m_activeFileName + " - topology/input params changed since last save; discarding incompatible saved weights and starting fresh");
FileDelete(m_activeFileName + ".nnw", m_activeFileCommon ? FILE_COMMON : 0);
}
if(FileIsExist(m_activeFileName + "_ckpt.tmp"))
FileDelete(m_activeFileName + "_ckpt.tmp");
// Same reasoning applies to the EMA shadow-weight file (see m_shadowNet's declaration comment) -
// it's shaped for the OLD topology too, and EnsureShadowNet() has no independent way to detect
// that mismatch on Load() (CNet::Load() doesn't cross-validate against an expected shape). Drop
// it so EnsureShadowNet() cleanly misses and re-bootstraps from the fresh Net instead.
if(FileIsExist(m_activeFileName + "_shadow.nnw", m_activeFileCommon ? FILE_COMMON : 0))
FileDelete(m_activeFileName + "_shadow.nnw", m_activeFileCommon ? FILE_COMMON : 0);
SaveTopologyConfiguration(m_activeFileName, m_initialNeuronsCount, m_hiddenLayersCount, m_neuronsReduction, m_minNeuronsCount, m_optimizationAlgo, m_historyBars, m_outputNeuronsCount, m_neuronsCount, m_studyPeriod, m_minTrainYear, m_isInitialized, m_stopTrainWR, m_fractalPeriods, m_activeFileCommon);
}
double loadedIndicatorParams[];
bool netLoaded = Net.Load(m_activeFileName + ".nnw", dError, dUndefine, dForecast, dtStudied, m_activeFileCommon, m_eraCount, m_trainingComplete, loadedIndicatorParams);
//--- a restart that recovers existing weights already has a proven-synced history and (since it
//--- already has at least one trained era behind it) doesn't have era 0's cold-start oversampling
//--- problem either - only a genuinely fresh start needs the 3 warm-up passes (see Train()'s
//--- m_warmupPassesRemaining gate).
//--- The label cache itself, however, is NEVER restored from the .nnw checkpoint - it lives only in
//--- the in-memory m_labelCacheBuy/Sell/HasValue arrays, which start empty every process start
//--- regardless of netLoaded. Previously this was set to netLoaded, which on a successful checkpoint
//--- load skipped the eager StartLabelCachePrebuild()/AdvanceLabelCachePrebuild() scan (Train()'s
//--- !m_labelCachePrebuilt gate) - every bar then fell through to the lazy per-bar fallback in the
//--- era loop, which calls ComputeLabelForBar(), a dead stub that unconditionally returns
//--- buy=false/sell=false (the real ZigZag-based labeling logic lives ONLY in
//--- AdvanceZigZagLabelState(), reachable exclusively from the eager prebuild). The result: every
//--- restart that loaded a checkpoint silently force-labeled the entire era Neutral until something
//--- else (a topology mismatch, a fresh start) triggered a real prebuild. Always eager-prebuilding
//--- now, checkpoint or not, closes this at the root - the dead stub fallback then never matters.
m_warmupPassesRemaining = netLoaded ? 0 : 3;
m_labelCachePrebuilt = false;
if(inTesterOrOpt && netLoaded)
Print(__FUNCTION__ + ": " + ID + " - reused cached weights from a previous optimization/tester pass on this agent (era " + IntegerToString(m_eraCount) + ", trainingComplete=" + (string)m_trainingComplete + ") - skipping redundant training for this unchanged config");
if(netLoaded && ArraySize(loadedIndicatorParams) == AD_TUNE_PARAM_COUNT)
{
// Restart deploying previously AutoTune-d indicator params even with AutoTuneIndicators=false now -
// indicators above were already created with today's defaults, so rebuild them once with the
// restored values before any training/signal work happens.
m_indicatorTuner.Unflatten(loadedIndicatorParams);
ReInitADIndicators(indicators);
}
if(!netLoaded)
{
int error_code = GetLastError();
if(error_code != 5004) // not "file not found"
{
printf("%s -> Error loading previous network %s error code : %d", __FUNCTION__, m_activeFileName, error_code);
ResetLastError();
}
if(!BuildFreshTopology())
return false;
}
TempData = new CArrayDouble();
if(CheckPointer(TempData) == POINTER_INVALID)
return false;
if(netLoaded)
// Populate dPrevSignal from the just-loaded weights immediately, rather than leaving it at
// its blank constructor default until the next (asynchronous, queued) training pass happens
// to run - matters most for the tester cache-reuse path above, where training may be skipped
// entirely for this run because dtStudied already covers the whole backtest window.
RefreshLatestSignal();
Print(__FUNCTION__ + ": " + m_activeFileName + " - training " + (m_trainingComplete ? "already complete - staying converged, no full retrain on this restart" : "NOT complete (interrupted or never converged) - resuming full training now"));
//--- Only kick off a full Train() run here if the loaded model genuinely isn't converged yet - an
//--- already-complete model used to get one full era-loop retrain (real Net.backProp() over the
//--- whole IS window) on every single EA restart/reattach for no reason, since this "Init" event
//--- bypassed ScheduleTrainingIfNeeded()'s m_trainingComplete gate entirely. dPrevSignal is already
//--- fresh from RefreshLatestSignal() above; ScheduleTrainingIfNeeded()'s normal per-tick check
//--- will call RefreshConvergedSignal() itself once a genuinely new bar closes.
if(!m_trainingComplete)
bEventStudy = EventChartCustom(ChartID(), 1, (long)MathMax(0, MathMin(iTime(_Symbol, PERIOD_CURRENT, (int)(100 * Net.recentAverageSmoothingFactor * (dForecast >= m_stopTrainWR ? 1 : 10))), dtStudied)), 0, "Init");
//--- bootstrap (or restore) the EMA shadow net now rather than waiting for the first
//--- RefreshLatestSignal()/era-blend call to lazily trigger it - see m_shadowNet's declaration
//--- comment.
EnsureShadowNet();
m_isInitialized = true;
return true;
}
//+------------------------------------------------------------------+
//| Builds a fresh, untrained topology into Net - the exact layer |
//| construction InitNeuralNetwork() used to inline for the |
//| "no saved .nnw" case; factored out so TuneIndicatorsAndTrain() can|
//| get a clean-slate Net per trial without touching indicator init. |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::BuildFreshTopology()
{
CArrayObj *Topology = new CArrayObj();
if(CheckPointer(Topology) == POINTER_INVALID)
return false;
//--- Input Layer
CLayerDescription *desc = new CLayerDescription();
if(CheckPointer(desc) == POINTER_INVALID)
{
delete Topology;
return false;
}
desc.count = m_historyBars * m_neuronsCount;
desc.type = defNeuron;
desc.activation = NONE;
desc.optimization = (ENUM_OPTIMIZATION)m_optimizationAlgo;
if(!Topology.Add(desc))
{
delete Topology;
return false;
}
//--- neuron-type-specific layers (Conv+Pool, LSTM, or none for a plain perceptron)
if(!AddCustomLayers(Topology))
{
delete Topology;
return false;
}
//--- Hidden Layers, tapering from m_initialNeuronsCount down to m_minNeuronsCount
int n = m_initialNeuronsCount;
bool result = true;
for(int i = 0; (i < m_hiddenLayersCount && result); i++)
{
desc = new CLayerDescription();
if(CheckPointer(desc) == POINTER_INVALID)
{
delete Topology;
return false;
}
desc.count = n;
desc.type = defNeuron;
desc.activation = HiddenLayerActivation();
desc.optimization = (ENUM_OPTIMIZATION)m_optimizationAlgo;
result = (Topology.Add(desc) && result);
// m_neuronsReduction is the reduction percentage itself (see NEURONS_REDUCTION_FACTOR's
// declaration comment), so retention is the complement, not the raw value.
n = (int)MathMax(n * ((100.0 - m_neuronsReduction) * 0.01), m_minNeuronsCount);
}
if(!result)
{
delete Topology;
return false;
}
//--- Output Layer
desc = new CLayerDescription();
if(CheckPointer(desc) == POINTER_INVALID)
{
delete Topology;
return false;
}
desc.count = m_outputNeuronsCount;
desc.type = defNeuron;
// Regression (1 output): TANH - its native [-1,1] range maps directly onto the -1/0/1
// Sell/Neutral/Buy target convention, without SIGMOID's [0,1] offset or ReLU's clipped
// negative half.
// Classification (3 outputs): SIGMOID, not NONE. The FORWARD activation is still SIGMOID (bounded,
// 0..1 per neuron) - NONE (raw, unbounded identity) used to sit here on the theory that softmax
// "needs" unbounded logits to work against, but with nothing to saturate it, class-balance
// oversampling replaying the same rare-class bar up to 5x in a row (see Train()'s reps loop) let
// Adam's momentum walk one neuron's raw output arbitrarily far in one direction - observed in
// practice as the reported IS error growing to ~2.5e29 within two eras and training permanently
// collapsing onto whichever class's logit happened to run away first. SIGMOID reuses the exact
// same bounded, clamped weight-update math (native MQL5, OpenCL, WarriorCPU.dll, WarriorDML.dll)
// the regression path already relies on, just applied per-neuron.
// The BACKWARD gradient, however, is NOT 3 independent per-neuron sigmoid deltas anymore -
// CNet::backProp()/backPropOCL() (AI\Network.mqh) detect the 3-output classification case and
// compute a true softmax+categorical-cross-entropy gradient (softmax_i - target_i) across all 3
// neurons jointly before the per-sample class-balance/focal-loss weighting is applied. This is
// what ties Buy/Sell/Neutral together during training - raising one class's probability now
// structurally lowers the other two via the shared softmax normalizer - closing the gap between
// "3 parallel binary regressions" (the old behavior, prone to jointly drifting toward all-Neutral)
// and a real mutually-exclusive multi-class objective, while keeping the bounded SIGMOID forward
// pass that fixed the logit-runaway collapse above. ApplyClassificationSoftmax() at read time now
// reproduces exactly the normalization the loss was trained against (previously it was purely a
// post-hoc display transform, decoupled from the actual training signal).
desc.activation = (m_outputNeuronsCount == 1) ? TANH : SIGMOID;
desc.optimization = (ENUM_OPTIMIZATION)m_optimizationAlgo;
if(!Topology.Add(desc))
{
delete Topology;
return false;
}
if(CheckPointer(Net) != POINTER_INVALID)
delete Net;
Net = new CNet(Topology);
delete Topology;
if(CheckPointer(Net) == POINTER_INVALID)
return false;
// A fresh topology invalidates any existing shadow (see m_shadowNet's declaration comment) -
// its weights, if any, are shaped for the OLD Net and would either mismatch dimensionally or,
// worse, silently blend unrelated weight spaces if the shape happens to coincide. Reset to NULL
// here; EnsureShadowNet() lazily re-bootstraps a fresh clone of the new Net on first use.
if(CheckPointer(m_shadowNet) != POINTER_INVALID)
{
delete m_shadowNet;
m_shadowNet = NULL;
}
return true;
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::InitIndicators(CIndicators *indicators)
{
PurgeChart();
if(!InitOpen(indicators))
return false;
if(!InitClose(indicators))
return false;
if(!InitLow(indicators))
return false;
if(!InitHigh(indicators))
return false;
//--- label source, always created unconditionally, same as the OHLC indicators above - see
//--- m_ADZigZag's declaration comment. Optionally ALSO read as an input feature (m_useSwingContext,
//--- below) using the same already-running indicator instance - no separate init needed for that.
if(!InitADZigZag(indicators))
return false;
m_neuronsCount = 4; // (close-open)/atr, (high-open)/atr, (low-open)/atr, bullish/bearish flag
if(m_useVolumes)
{
m_neuronsCount++;
if(!InitVolumes(indicators))
return false;
}
if(m_useTime)
{
m_neuronsCount += 6;
if(!InitTime(indicators))
return false;
}
if(m_useATR)
{
//already init in the base class
m_neuronsCount++;
}
if(m_useSwingContext)
m_neuronsCount += 5; // direction, distance-since-pivot, prior-leg magnitude, retracement ratio, bars-since-pivot - see BufferTempDataCompute()'s matching block
if(m_useNews)
m_neuronsCount += 2; // NewsRecency, NewsProximity - see BufferTempDataCompute()'s matching block
if(m_useADCumulativeDelta)
{
if(!InitADCumulativeDelta(indicators))
return false;
m_neuronsCount += 6; // Pressure, CumulativeDelta, BullishPressure, BearishPressure, Absorption, Initiative
}
if(m_useADShorteningOfThrust)
{
if(!InitADShorteningOfThrust(indicators))
return false;
m_neuronsCount += 4; // SOT, SOTEffortRegime, SOTConfirmation, SOTPushRegime
}
if(m_useADWyckoffEventStream)
{
if(!InitADWyckoffEventStream(indicators))
return false;
// 12, not 14 - EventPrice (buffer 4) and EventPhase (buffer 1) are deliberately excluded, see
// BufferTempData()'s comment
m_neuronsCount += 12; // EventCode, ZoneTop, ZoneBottom, StructuralPhase, CHoCHTrendToRange, CHoCHRangeToTrend, SlopeAccumulationBullish, SlopeAccumulationBearish, SlopeDistributionBullish, SlopeDistributionBearish, Reaccumulation, Redistribution
}
if(m_useADWyckoffFailedStructure)
{
if(!InitADWyckoffFailedStructure(indicators))
return false;
m_neuronsCount += 5; // Value, BullishStructuralFailure, BearishStructuralFailure, FailedAccumulation, FailedDistribution
}
if(m_useADWyckoffSignificantBarInversion)
{
if(!InitADWyckoffSignificantBarInversion(indicators))
return false;
m_neuronsCount += 5; // SignificantBarQuality, BullishSignificantBar, BearishSignificantBar, BullishControlFlip, BearishControlFlip
}
if(!FolderCreate(m_folderPath, FILE_COMMON))
{
if(GetLastError() != 5010) // If the error is not because the folder already exists
{
Print("Failed to create folder: " + m_folderPath);
}
else
{
ResetLastError(); // Reset the error code
}
}
return true;
}
//+------------------------------------------------------------------+
//| Training and Signal Methods |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::Train(datetime StartTrainBar = 0)
{
const int STABILITY_WINDOW = 3; // consecutive eras the OOS accuracy must hold steady for
const double STABILITY_TOLERANCE = 2.0; // max spread (percentage points) across that window
// Max wall-clock work per call before yielding - see m_trainRunActive's declaration comment for
// why chunking exists at all. TuneIndicatorsAndTrain()/Train() only run ONCE per dispatched
// "New Bar" custom chart event (see OnChartEventHandler - id 1001 calls it exactly once, then
// clears bEventStudy so ScheduleTrainingIfNeeded() can arm the next one), so the real throughput
// ceiling in practice is however fast MT5 itself pumps/dispatches that custom event - NOT this
// constant. Raising the OnTimer interval (5s->250ms) had ~zero effect for exactly that reason:
// ticks/chart events were already redispatching far more often than the timer alone would. Since
// per-event dispatch overhead is roughly fixed, doing more compute per event (fewer, larger
// chunks) cuts wall-clock training time roughly in proportion, but MT5 has only this one thread -
// the panel/chart can only respond to input in the gap between chunks, so 500ms made it feel
// unresponsive unless clicks landed in that narrow window. Lowered back to 120ms to keep the UI
// reactive; raise again only if throughput becomes the bigger complaint.
const uint TRAIN_TIME_BUDGET_MS = 120;
//---
//--- Never block the calling thread while paused/stopped - just decline this call (or finalize a
//--- run that just got stopped) and let the next scheduled call check again, so Pause/Resume/Stop
//--- and everything else on the control panel stays responsive instead of Sleep()-ing the one
//--- MQL5 thread this chart has.
if(m_trainingPaused && !IsStopped() && !m_trainingStopRequested)
return;
bool stop = IsStopped() || m_trainingStopRequested;
if(stop)
{
if(m_trainRunActive)
FinalizeTrainRun();
if(m_simOosRunActive)
{
delete m_simOosNet;
m_simOosNet = NULL;
m_simOosRunActive = false;
}
return;
}
//--- Evaluation-only continual-learning OOS simulation walk in progress (see StartOosContinualSimulation):
//--- give it exclusive occupancy of this call, same chunked budget as the real era loop below, so a
//--- large OOS window can't freeze the UI in one shot. While it's active no real-training
//--- ResizeBuffers()/RefreshData() runs, so the price/ATR/time buffers it reads stay frozen for its
//--- whole walk - it never has to worry about the label cache's shifting-index invalidation below.
if(m_simOosRunActive)
{
AdvanceOosSimulationChunk();
return;
}
//--- Eager label-cache pre-build in progress (see StartLabelCachePrebuild/AdvanceLabelCachePrebuild) -
//--- same exclusive-occupancy/chunking treatment as the OOS simulation walk above, so it can't freeze
//--- the UI on a large study window either. m_trainRunActive stays false for its whole duration, so
//--- once it completes, Train() falls through to the normal !m_trainRunActive setup below and era 0
//--- starts with the oversampling ratio it just seeded.
if(m_labelPrebuildActive)
{
AdvanceLabelCachePrebuild();
return;
}
if(!m_trainRunActive)
{
//--- Wait (briefly, bounded, non-blocking across calls) for the terminal to finish syncing this
//--- symbol/period's history from the broker before computing the training window. Bars(symbol,
//--- period) - the hard cap on how many bars the era loop below will ever process - reflects
//--- whatever's synced SO FAR, not necessarily the true total; starting before sync completes
//--- would let that cap (and therefore the "Bar X of Y" progress display) silently grow between
//--- eras as more history trickles in.
if(!SeriesInfoInteger(m_symbol.Name(), PERIOD_CURRENT, SERIES_SYNCHRONIZED))
{
uint syncNowTick = GetTickCount();
if(m_syncWaitStartTick == 0)
m_syncWaitStartTick = syncNowTick;
if(syncNowTick - m_syncWaitStartTick < 5000)
return; // retry on the next scheduled call instead of blocking here
Print(ID + ": WARNING - history for " + m_symbol.Name() + " " + EnumToString(PERIOD_CURRENT) + " did not finish syncing after 5s; training window may still grow as more history arrives");
}
m_syncWaitStartTick = 0;
//--- 3 no-op passes before the era loop ever runs for a fresh start (see m_warmupPassesRemaining's
//--- declaration comment) - each is its own separately-scheduled Train() call (this whole method
//--- just returns, deferring to the next "New Bar"/timer-driven call), giving MT5's history sync
//--- several real, wall-clock-separated chances to settle on top of the 5s soft wait just above,
//--- before training commits to a bar count and starts populating the label cache below.
if(m_warmupPassesRemaining > 0)
{
m_warmupPassesRemaining--;
PrintVerbose(ID + ": warm-up pass " + IntegerToString(3 - m_warmupPassesRemaining) + " of 3 (letting history sync settle before training starts)");
return;
}
MqlDateTime start_time;
TimeCurrent(start_time);
start_time.year -= m_studyPeriod;
if(start_time.year <= 0 || start_time.year < m_minTrainYear)
start_time.year = m_minTrainYear;
datetime st_time = StructToTime(start_time);
dtStudied = MathMax(StartTrainBar, st_time);
//--- Clamp the requested training window to what actually exists: a StudyPeriod (or MinTrainYear)
//--- reaching further back than the symbol's real history (e.g. broker feed only starts 2019, but
//--- StudyPeriod=20 asks for 20 years back from today) must not change how much data training
//--- thinks it has to cover - only the genuinely available bars ever get processed regardless, so
//--- clamping here just keeps the displayed window/progress honest instead of implying a longer
//--- span than actually exists.
datetime firstAvailableBar = (datetime)SeriesInfoInteger(m_symbol.Name(), PERIOD_CURRENT, SERIES_FIRSTDATE);
if(firstAvailableBar > 0 && dtStudied < firstAvailableBar)
dtStudied = firstAvailableBar;
//--- OOS-based objective + stability tracking: training only "converges" once the objective is
//--- met AND OOS accuracy has held inside a tight band for the last few eras, so a single lucky
//--- era can't get locked in as the final model. The best-scoring era's weights are checkpointed
//--- to an agent-local scratch file (not FILE_COMMON) and restored at the end - this works inside
//--- the tester too, unlike Net.Save()/Load() which are disabled there.
m_oosWindow.Clear();
m_bestOosForecast = -1;
m_bestPassedRecall = false;
m_haveOosCheckpoint = false;
m_oosStable = false;
m_objectiveMet = false;
m_erasSinceCooldown = 0;
m_eraResumePending = false;
//--- One-time eager pre-scan for a fresh start (see m_labelCachePrebuilt's declaration comment) -
//--- kick it off and defer era 0 until it's done, so era 0 can start with a real class-balance
//--- oversampling ratio instead of the reps=1 fallback. Routed via the m_labelPrebuildActive gate
//--- above on every subsequent call until it completes.
if(!m_labelCachePrebuilt)
{
StartLabelCachePrebuild();
return;
}
m_trainRunActive = true;
}
int bars, totalIter, oosCutoff, i;
bool add_loop;
if(!m_eraResumePending)
{
int barsNow = (int)MathMin(Bars(m_symbol.Name(), PERIOD_CURRENT, dtStudied, TimeCurrent()) + m_historyBars, Bars(m_symbol.Name(), PERIOD_CURRENT));
if(!ResizeBuffers(barsNow) || !RefreshData())
{
FinalizeTrainRun();
return;
}
bars = barsNow;
add_loop = false;
//--- Label/feature cache invalidation: MQL5 timeseries indices are always relative to "now"
//--- (index 0 = current bar), so every new closed candle shifts every older bar's index - a
//--- cache keyed by index would silently misalign the moment that happens. See
//--- EnsureBarCachesCapacity() for why `bars` + m_Time.GetData(0) are the correct/sufficient
//--- invalidation keys.
EnsureBarCachesCapacity(bars);
//--- freeze the just-finished era's true class totals for this new era's oversampling ratio (see
//--- m_prevEraTrueBuyCount's declaration comment) before resetting the live counters below - EXCEPT
//--- right after StartLabelCachePrebuild()/AdvanceLabelCachePrebuild() seeded them for era 0: the
//--- live m_trueBuyCount/Sell/Neutral tally is still all-zero at that point (nothing trained yet),
//--- so copying it here would silently stomp the real upfront tally right back to reps=1.
if(m_prebuildSeedPending)
m_prebuildSeedPending = false;
else
{
m_prevEraTrueBuyCount = m_trueBuyCount;
m_prevEraTrueSellCount = m_trueSellCount;
m_prevEraTrueNeutralCount = m_trueNeutralCount;
}
m_countBuySignals = 0;
m_countSellSignals = 0;
m_countNeutralSignals = 0;
m_trueBuyCount = 0;
m_trueSellCount = 0;
m_trueNeutralCount = 0;
m_oosBuyHits = 0;
m_oosBuyTotal = 0;
m_oosSellHits = 0;
m_oosSellTotal = 0;
m_oosNeutralHits = 0;
m_oosNeutralTotal = 0;
m_oosBuyPredicted = 0;
m_oosBuyPredictedHits = 0;
m_oosSellPredicted = 0;
m_oosSellPredictedHits = 0;
m_oosNeutralPredicted = 0;
m_oosNeutralPredictedHits = 0;
m_oosConfidenceSum = 0;
// Nearest-to-present slice of this era's bars is held out as OOS and never backprop'd on;
// the rest (older bars) is the IS/training slice.
totalIter = (int)MathMax(bars - MathMax(m_historyBars, 0), 0);
oosCutoff = (int)(MathMax(0, MathMin(100, m_oosSplitPct)) / 100.0 * totalIter);
i = (int)(bars - MathMax(m_historyBars, 0) - 1);
//--- Fresh era: reset pass 2's shuffled-backprop queue (see m_isTrainQueue's declaration
//--- comment) - preallocated to totalIter * MAX_OVERSAMPLE_REPLICAS, a safe upper bound even in
//--- the (never actually reachable) worst case of every single IS-eligible bar this era being a
//--- minority class replicated the maximum number of times - see the queueing block below for
//--- where the real (much smaller) per-bar replica count actually gets decided.
ArrayResize(m_isTrainQueue, totalIter * MAX_OVERSAMPLE_REPLICAS);
ArrayResize(m_isTrainQueueWeightScale, totalIter * MAX_OVERSAMPLE_REPLICAS);
m_isTrainQueueCount = 0;
m_isTrainCursor = 0;
m_isPass2Active = false;
m_isPass2Done = false;
m_isPass3Active = false;
}
else
{
//--- resuming a chunk that yielded mid-bar-loop last call - pick up exactly where it left off
bars = m_resumeBars;
totalIter = m_resumeTotalIter;
oosCutoff = m_resumeOosCutoff;
add_loop = m_resumeAddLoop;
i = m_resumeBarIndex;
m_eraResumePending = false;
}
// Restore this model's own learning-rate trajectory into the shared global right before this
// chunk's backProp() calls touch it - see m_modelEta's declaration comment.
eta = m_modelEta;
uint chunkStartTick = GetTickCount();
// Iterate over the bars - skipped entirely when resuming straight into pass 2, OR when resuming
// into a still-unfinished pass 3 (see m_isPass2Done's declaration comment for why checking
// m_isPass2Active alone isn't enough to detect the latter case): pass 1 already fully completed
// in an earlier call either way.
if(!m_isPass2Active && !m_isPass2Done)
{
for(; i >= 0 && !stop; i--)
{
//--- Build THIS bar's own feature window and feed it forward BEFORE checking/training against
//--- its label - see r's declaration comment below for why the window must end AT bar i, and
//--- why this must run before the label-check block rather than after: the label check needs
//--- this bar's own freshly-computed prediction, not the previous iteration's (see windowOk).
TempData.Clear();
TempData.Reserve((int)m_historyBars * m_neuronsCount);
//--- Window ends AT (includes) bar i itself, extending m_historyBars bars into the past - i.e.
//--- "everything known as of this bar's close." Predicting label(i) - "was THIS bar the
//--- reversal" - from a window that stops short of bar i itself would blind the model to the
//--- most recent price action, which is exactly the information a reversal call most depends
//--- on. Must match RefreshLatestSignal()'s window exactly (r=i there too, i=0), since that's
//--- what actually queries the deployed model live - training on a different window than what
//--- gets queried at inference time would teach the wrong task entirely.
int r = i;
bool windowOk = false;
double displayNeuron0 = 0, displayNeuron1 = 0, displayNeuron2 = 0;
if(r <= bars)
{
for(int b = 0; b < (int)m_historyBars; b++)
{
int bar_t = r + b;
if(!BufferTempData(bar_t))
break;
}
if(TempData.Total() >= (int)m_historyBars * m_neuronsCount)
{
windowOk = true;
add_loop = true;
}
}
//--- Determine label/queue-eligibility BEFORE running any feedForward this bar - see
//--- wouldQueue's use below for why. Mirrors the label-check condition this block used to
//--- gate on (moved earlier, unchanged).
bool haveLabel = false, buy = false, sell = false, wouldQueue = false;
if(windowOk && i < (int)(bars - MathMax(m_historyBars, 0) - 1) && i > 1 && m_Time.GetData(i) > dtStudied
&& (m_outputNeuronsCount == 1 || m_outputNeuronsCount == 3))
{
//--- The fractal/swing-confirmation/trend-context label at now-relative index i only depends
//--- on price/ATR history, never on model state, so it's identical every era until a new bar
//--- closes and shifts the index frame (see the cache invalidation check above) - cache it
//--- rather than recomputing from scratch every single era. Usually already populated by
//--- AdvanceLabelCachePrebuild() before era 0 ever starts - this is just a lazy fallback for
//--- any index it didn't cover (e.g. bars/window drifted between prebuild and era 0's start).
if(m_labelCacheHasValue[i])
{
buy = m_labelCacheBuy[i];
sell = m_labelCacheSell[i];
}
else
{
ComputeLabelForBar(i, bars, buy, sell);
m_labelCacheBuy[i] = buy;
m_labelCacheSell[i] = sell;
m_labelCacheHasValue[i] = true;
}
haveLabel = true;
bool isOOS = (i < oosCutoff);
// Embargo: a ZigZag pivot at bar `idx` is only confirmed once m_swingConfirmationBars
// MORE (more-recent) bars have closed after it (see AdvanceZigZagLabelState()'s
// declaration comment), further widened by +-LABEL_WINDOW_BARS. An IS bar within that
// distance of the OOS boundary can therefore carry a label that was only knowable using
// price action from inside the held-out OOS window - purge that narrow band from
// backprop entirely instead of training on it as ordinary IS.
int embargoBars = m_swingConfirmationBars + LABEL_WINDOW_BARS;
bool isEmbargoed = (!isOOS && i < oosCutoff + embargoBars);
wouldQueue = (!isOOS && !isEmbargoed);
}
//--- Only run this bar's feedForward (and the display/count/chart-draw work that depends on
//--- it) when pass 2 ISN'T about to redo it anyway. A queued bar gets a completely fresh
//--- feedForward moments later in pass 2 (see m_isTrainQueue's declaration comment - other
//--- queued bars ahead of it in the shuffled order may already have updated weights, so pass
//--- 2 can't reuse this scan's result even if it wanted to) - running it here too was one
//--- full forward pass per training sample thrown straight in the trash every era, on top of
//--- the one pass 2 actually needs. The book's SGD (references\neuronetworksbook.pdf, section
//--- 1.4) is one forward+backward pass per training sample, not two.
if(windowOk && !wouldQueue)
{
Net.feedForward(TempData);
Net.getResults(TempData);
if(m_outputNeuronsCount == 1)
dPrevSignal = TempData[0];
else
if(m_outputNeuronsCount == 3)
dPrevSignal = ApplyClassificationSoftmax();
//--- Snapshot the just-computed neuron output(s) for the status label display below, before
//--- the label-check block clears/refills TempData with the target label (Step A always
//--- runs after this point now) - reading TempData directly for display after that would
//--- show the TRUE LABEL of the bar just trained on, not the network's own prediction.
if(TempData.Total() > 0)
displayNeuron0 = TempData[0];
if(TempData.Total() > 1)
displayNeuron1 = TempData[1];
if(TempData.Total() > 2)
displayNeuron2 = TempData[2];
switch(DoubleToSignal(dPrevSignal))
{
case Buy:
m_countBuySignals++;
break;
case Sell:
m_countSellSignals++;
break;
default:
m_countNeutralSignals++;
break;
}
m_lastBarTime = m_Time.GetData(i);
if(i > 0)
{
if(DoubleToSignal(dPrevSignal) == Neutral)
DeleteObject(m_lastBarTime);
else
DrawObject(m_lastBarTime, dPrevSignal, m_High.GetData(i), m_Low.GetData(i));
}
UpdateTrainingStatusLabel(
StringFormat("Bar %d of %d -> %.2f%% (scan)", bars - i + 1, bars, (double)(bars - i + 1.0) / bars * 100),
displayNeuron0, displayNeuron1, displayNeuron2, dPrevSignal);
}
if(haveLabel)
{
// True label as an ENUM_SIGNAL, derived directly from the buy/sell bools - not read
// back from TempData, which no longer holds a target at this point at all (see above).
ENUM_SIGNAL trueSignal = buy ? Buy : (sell ? Sell : Neutral);
// Track the true class distribution this era (used below to weight IS oversampling,
// and surfaced in the status label text alongside the predicted-class counts)
switch(trueSignal)
{
case Buy:
m_trueBuyCount++;
break;
case Sell:
m_trueSellCount++;
break;
default:
m_trueNeutralCount++;
break;
}
// OOS scoring used to happen right here, against whatever weights this bar's earlier
// feedForward (this pass) happened to be using - which for era 0 is the network's
// still-untrained cold-start state (100% Neutral - see the output-layer bias seed's
// declaration comment), and for every later era is last era's END-of-training state,
// never THIS era's. That silently gave every era's OOS score a full one-era lag behind
// its own training, and made era 0's OOS score meaningless by construction. OOS scoring
// now happens in its own pass (see m_isPass3Active's declaration comment), AFTER pass 2
// has actually trained on this era's IS data, against a fresh feedForward on each OOS
// bar rather than this scan's now-stale one.
if(wouldQueue)
{
// Queue this bar for pass 2's shuffled backProp instead of training on it here,
// immediately, in strict chronological order - see m_isTrainQueue's declaration
// comment for the full rationale. Class-balance weighting and focal-loss
// modulation (m_maxClassSampleWeight/m_focalGamma), the predicted-signal counts,
// the chart-marker draw, and the dForecast/dUndefine IS-accuracy update are all
// computed in pass 2 instead now, against that bar's own freshly-recomputed
// confidence - see the matching block right after pass 2's Net.feedForward() call.
//
// Data-level oversampling: a minority-class (Buy/Sell) bar gets queued more than once
// (capped at MAX_OVERSAMPLE_REPLICAS), using the same previous-era class ratio
// m_maxClassSampleWeight's loss weight is derived from. Reason this exists alongside
// that loss weight rather than instead of it: sampleWeight only scales the gradient
// AFTER it's computed, and Adam's update (AI\Network.mqh's UpdateWeightsAdam, lt *
// mt/sqrt(vt)) is deliberately near-invariant to a constant rescaling of the gradient
// (Kingma & Ba 2015) - mt and vt both move with the weight, so the ratio largely
// cancels it back out, which is why a purely loss-weighted 5x+ imbalance was observed
// in practice to plateau at a full Neutral-only collapse for 100+ eras instead of
// correcting (era 20-115 of one run stuck at OOS accuracy ~73.6%, Buy/Sell recall 0%).
// Duplicating the bar itself changes how often Adam's moment estimates see a Buy/Sell
// gradient at all, which the mt/sqrt(vt) normalization can't cancel out the same way.
//
// repCount below is driven by the RAW previous-era class ratio (uncapped by
// m_maxClassSampleWeight) - the actual imbalance is what needs correcting, not a
// pre-shrunk version of it - bounded only by MAX_OVERSAMPLE_REPLICAS for the
// momentum-correlation safety reason above. perOccurrenceScale is pinned at 1.0:
// repCount's duplication IS the whole correction now, not one half of a split budget.
// Two earlier versions of this both stacked a SECOND multiplicative correction on top
// of repCount and both collapsed, in opposite directions: v1 let pass 2 independently
// reach for its own full m_maxClassSampleWeight on top of an uncapped repCount (up to
// 3 x 1.5 = 4.5x total), which overshot into a Buy-only (or Sell-only) collapse -
// OOS accuracy dropping to ~10%, IS error climbing 0.37->0.57 over 4 eras, the same
// Adam-overshoot signature documented at AI\Network.mqh's MAX_WEIGHT_DELTA comment,
// just from a different cause. v2 tried to "coordinate" the split by capping the ratio
// at m_maxClassSampleWeight BEFORE dividing it between repCount and perOccurrenceScale
// - but repCount x perOccurrenceScale is then mathematically forced to equal that same
// <=1.5x ceiling no matter how it's split, i.e. IDENTICAL total correction magnitude to
// the pure loss-weighting that already failed to escape the original Neutral collapse
// (era 20-115, OOS ~73.6%, Buy/Sell recall 0%) - so it just silently undershot back
// into the same failure it was meant to fix. Per Buda, Maki & Mazurowski 2018 (Neural
// Networks), combining data-level oversampling with cost-sensitive loss reweighting on
// the SAME class-balance axis doesn't reliably help and can hurt; oversampling alone is
// competitive or better. v3 let repCount alone carry the correction but capped
// MAX_OVERSAMPLE_REPLICAS at 3 - only a ~1.75-1.8x residual imbalance against the
// observed ~5.2-5.3x ratio - and still collapsed the same way (OOS accuracy climbing
// toward the ~73.6% Neutral base rate while Buy/Sell recall stayed 0% for the first 6
// eras straight, 2026-07-18). The double-counting removal was correct; the cap just
// wasn't raised far enough to reach it. MAX_OVERSAMPLE_REPLICAS is now 5, letting
// repCount reach much closer to full class parity for this ratio, per Buda et al.'s
// finding that oversampling to parity is safe.
//
// Queued BEFORE pass 2's Fisher-Yates shuffle (which runs on the whole queue, repeats
// included), so duplicates land scattered through the era's shuffled replay order, not
// back-to-back - avoiding the correlated-momentum Adam-overshoot bug a cruder
// back-to-back oversampling replay caused historically (see AI\Network.mqh's
// MAX_WEIGHT_DELTA comment).
double rawRatio = 1.0;
if(trueSignal != Neutral)
{
int prevMaxCount = (int)MathMax(m_prevEraTrueBuyCount, MathMax(m_prevEraTrueSellCount, m_prevEraTrueNeutralCount));
int prevTrueCount = (trueSignal == Buy) ? m_prevEraTrueBuyCount : m_prevEraTrueSellCount;
if(prevTrueCount > 0 && prevMaxCount > 0)
rawRatio = (double)prevMaxCount / prevTrueCount;
}
int repCount = (int)MathMin(MAX_OVERSAMPLE_REPLICAS, MathMax(1, MathRound(rawRatio)));
double perOccurrenceScale = 1.0;
for(int rep = 0; rep < repCount; rep++)
if(m_isTrainQueueCount < ArraySize(m_isTrainQueue))
{
m_isTrainQueue[m_isTrainQueueCount] = i;
m_isTrainQueueWeightScale[m_isTrainQueueCount] = perOccurrenceScale;
m_isTrainQueueCount++;
}
}
}
stop = IsStopped() || m_trainingStopRequested;
if(!stop && i > 0 && GetTickCount() - chunkStartTick >= TRAIN_TIME_BUDGET_MS)
{
//--- yield: save exactly enough to resume this same era, mid-bar-loop, on the next call -
//--- see m_trainRunActive's declaration comment for why this must happen instead of
//--- letting one era (or the whole run) process synchronously to completion
m_resumeBars = bars;
m_resumeTotalIter = totalIter;
m_resumeOosCutoff = oosCutoff;
m_resumeAddLoop = add_loop;
m_resumeBarIndex = i - 1;
m_eraResumePending = true;
// Save this model's own learning-rate trajectory back out of the shared global before
// yielding - see m_modelEta's declaration comment.
m_modelEta = eta;
return;
}
}
} // end if(!m_isPass2Active) - pass 1
//--- Pass 2: replay the bars pass 1 queued into m_isTrainQueue for backProp, in a freshly
//--- shuffled order - see m_isTrainQueue's declaration comment for the full rationale. Runs
//--- whenever pass 1 just finished (or we resumed straight into an already-active pass 2 - see
//--- m_isPass2Active's declaration comment); skipped on a stopped run, an era with no valid window
//--- at all (add_loop still false), or - critically - a resume into a still-unfinished pass 3 (see
//--- m_isPass2Done's declaration comment): without this last check, that resume would re-shuffle
//--- and replay the ENTIRE queue again from scratch every single call.
if(!stop && add_loop && !m_isPass2Done)
{
if(!m_isPass2Active)
{
m_isPass2Active = true;
m_isTrainCursor = 0;
// Fisher-Yates shuffle - a fresh random order every era, so Adam's momentum can't keep
// landing on the same short same-class label run (see LABEL_WINDOW_BARS) at the same point
// in the sequence every single era. m_isTrainQueueWeightScale is swapped in lockstep - each
// slot's stored per-occurrence weight (see the queueing block's oversampling comment) must
// stay attached to the same bar index it was computed for.
for(int sIdx = m_isTrainQueueCount - 1; sIdx > 0; sIdx--)
{
int sJ = MathRand() % (sIdx + 1);
int sTmp = m_isTrainQueue[sIdx];
m_isTrainQueue[sIdx] = m_isTrainQueue[sJ];
m_isTrainQueue[sJ] = sTmp;
double sScaleTmp = m_isTrainQueueWeightScale[sIdx];
m_isTrainQueueWeightScale[sIdx] = m_isTrainQueueWeightScale[sJ];
m_isTrainQueueWeightScale[sJ] = sScaleTmp;
}
}
for(; m_isTrainCursor < m_isTrainQueueCount; m_isTrainCursor++)
{
int qi = m_isTrainQueue[m_isTrainCursor];
TempData.Clear();
TempData.Reserve((int)m_historyBars * m_neuronsCount);
bool qWindowOk = true;
for(int b = 0; b < (int)m_historyBars; b++)
if(!BufferTempData(qi + b))
{
qWindowOk = false;
break;
}
if(qWindowOk && TempData.Total() >= (int)m_historyBars * m_neuronsCount)
{
Net.feedForward(TempData);
Net.getResults(TempData);
// Must go through ApplyClassificationSoftmax() (3-output case) before reading the
// per-class values below - Net.getResults() returns each output neuron's own independent
// SIGMOID activation (each already in [0,1] but NOT summing to 1 across the three), not a
// true class-conditional probability distribution; ApplyClassificationSoftmax() is what
// turns that into one (and is also what pass 1/3's displayNeuron0/1/2 already go through).
double qPrevSignal = (m_outputNeuronsCount == 3) ? ApplyClassificationSoftmax() : TempData[0];
double pt0 = (TempData.Total() > 0) ? TempData[0] : 0.0;
double pt1 = (TempData.Total() > 1) ? TempData[1] : 0.0;
double pt2 = (TempData.Total() > 2) ? TempData[2] : 0.0;
bool qBuy = m_labelCacheHasValue[qi] ? m_labelCacheBuy[qi] : false;
bool qSell = m_labelCacheHasValue[qi] ? m_labelCacheSell[qi] : false;
ENUM_SIGNAL qTrueSignal = qBuy ? Buy : (qSell ? Sell : Neutral);
UpdateTrainingStatusLabel(
StringFormat("Training bar %d of %d -> %.2f%% (shuffled)", m_isTrainCursor + 1, m_isTrainQueueCount, (double)(m_isTrainCursor + 1.0) / MathMax(m_isTrainQueueCount, 1) * 100),
pt0, pt1, pt2, qPrevSignal);
//--- Predicted-signal tally, chart marker, and IS-accuracy stat that pass 1 used to compute
//--- from its own (now-removed) redundant feedForward on this same bar - see pass 1's
//--- wouldQueue comment. Uses THIS feedForward's result (the only one this bar gets), so
//--- these now reflect the model's state as of this bar's own turn in the shuffled replay
//--- (post any earlier-shuffled bar's backProp this era), not a separate pre-training
//--- snapshot - matching how a standard shuffled-epoch SGD run reports running training
//--- accuracy during the epoch rather than in a discarded pre-epoch dry run.
switch(DoubleToSignal(qPrevSignal))
{
case Buy:
m_countBuySignals++;
break;
case Sell:
m_countSellSignals++;
break;
default:
m_countNeutralSignals++;
break;
}
datetime qBarTime = m_Time.GetData(qi);
if(DoubleToSignal(qPrevSignal) == Neutral)
DeleteObject(qBarTime);
else
DrawObject(qBarTime, qPrevSignal, m_High.GetData(qi), m_Low.GetData(qi));
bool qClassified = (DoubleToSignal(qPrevSignal) == Buy || DoubleToSignal(qPrevSignal) == Sell || DoubleToSignal(qPrevSignal) == Neutral);
if(qClassified)
{
if(DoubleToSignal(qPrevSignal) == qTrueSignal)
dForecast += (100 - dForecast) / Net.recentAverageSmoothingFactor;
else
dForecast -= dForecast / Net.recentAverageSmoothingFactor;
dUndefine -= dUndefine / Net.recentAverageSmoothingFactor;
}
else
if(qBuy && qSell)
dUndefine += (100 - dUndefine) / Net.recentAverageSmoothingFactor;
TempData.Clear();
if(m_outputNeuronsCount == 1)
TempData.Add(qBuy && !qSell ? 1 : !qBuy && qSell ? -1 : 0);
else
if(m_outputNeuronsCount == 3)
{
TempData.Add(qBuy ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW);
TempData.Add(qSell ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW);
TempData.Add((!qBuy && !qSell) ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW);
}
// Class-balance component comes from m_isTrainQueueWeightScale[m_isTrainCursor] - decided
// once at queue time in pass 1 (see that block's oversampling comment). Currently always
// 1.0: repCount's duplication is the entire class-balance correction now, so this array
// carries no additional per-occurrence weight on top of it - see the pass-1 comment for
// why stacking a second correction here caused two separate collapses. Kept as a real
// per-slot value (not a literal 1.0 inline) so a future, smaller/safer supplemental weight
// can be reintroduced here without re-touching the queueing or shuffle code.
// Focal-loss modulation is still applied fresh here though - see m_focalGamma's
// declaration comment. pt is this bar's OWN just-recomputed softmax confidence in its true
// class - fresh as of THIS pass-2 step, not pass 1's now-stale snapshot, since other
// queued bars ahead of it in the shuffled order may already have updated weights.
double qSampleWeight = m_isTrainQueueWeightScale[m_isTrainCursor];
if(m_outputNeuronsCount == 3 && m_focalGamma > 0.0)
{
double pt = (qTrueSignal == Buy) ? pt0 : (qTrueSignal == Sell) ? pt1 : pt2;
pt = MathMax(0.0, MathMin(1.0, pt));
qSampleWeight *= MathPow(1.0 - pt, m_focalGamma);
}
Net.backProp(TempData, qSampleWeight);
}
if(m_isTrainCursor + 1 < m_isTrainQueueCount && GetTickCount() - chunkStartTick >= TRAIN_TIME_BUDGET_MS)
{
//--- yield: save enough to resume PASS 2 mid-queue on the next call - m_isPass2Active
//--- and m_isTrainCursor (both members) carry the actual resume position; bars/oosCutoff/
//--- add_loop are stashed the same way pass 1 already does, since era-end logic just
//--- below still needs them once pass 2 finishes.
m_resumeBars = bars;
m_resumeTotalIter = totalIter;
m_resumeOosCutoff = oosCutoff;
m_resumeAddLoop = add_loop;
m_resumeBarIndex = i;
m_eraResumePending = true;
m_modelEta = eta;
return;
}
}
m_isPass2Active = false;
m_isPass2Done = true;
}
//--- Pass 3: OOS scoring, chronological, AFTER pass 2 has actually trained on this era's IS data -
//--- see m_isPass3Active's declaration comment for why this can no longer happen inline during
//--- pass 1's scan.
if(!stop && add_loop)
{
if(!m_isPass3Active)
{
m_isPass3Active = true;
m_oosScoreStartIndex = (int)MathMin(oosCutoff - 1, bars - MathMax(m_historyBars, 0) - 2);
m_oosScoreIndex = m_oosScoreStartIndex;
}
for(; m_oosScoreIndex >= 2; m_oosScoreIndex--)
{
int oi = m_oosScoreIndex;
if(!(oi < (int)(bars - MathMax(m_historyBars, 0) - 1) && m_Time.GetData(oi) > dtStudied))
continue;
TempData.Clear();
TempData.Reserve((int)m_historyBars * m_neuronsCount);
bool oWindowOk = true;
for(int b = 0; b < (int)m_historyBars; b++)
if(!BufferTempData(oi + b))
{
oWindowOk = false;
break;
}
if(oWindowOk && TempData.Total() >= (int)m_historyBars * m_neuronsCount)
{
Net.feedForward(TempData);
Net.getResults(TempData);
double oPrevSignal = (m_outputNeuronsCount == 3) ? ApplyClassificationSoftmax() : TempData[0];
double oNeuron0 = (TempData.Total() > 0) ? TempData[0] : 0.0;
double oNeuron1 = (TempData.Total() > 1) ? TempData[1] : 0.0;
double oNeuron2 = (TempData.Total() > 2) ? TempData[2] : 0.0;
bool oBuy = m_labelCacheHasValue[oi] ? m_labelCacheBuy[oi] : false;
bool oSell = m_labelCacheHasValue[oi] ? m_labelCacheSell[oi] : false;
ENUM_SIGNAL oTrueSignal = oBuy ? Buy : (oSell ? Sell : Neutral);
UpdateTrainingStatusLabel(
StringFormat("Scoring OOS bar %d of %d -> %.2f%% (post-training)", m_oosScoreStartIndex - m_oosScoreIndex + 1, m_oosScoreStartIndex + 1,
(double)(m_oosScoreStartIndex - m_oosScoreIndex + 1.0) / MathMax(m_oosScoreStartIndex + 1, 1) * 100),
oNeuron0, oNeuron1, oNeuron2, oPrevSignal);
// Held-out bar: score the model's freshly-trained-this-era forecast against the actual
// outcome without learning from it - keeps the OOS accuracy an honest overfitting signal.
bool oClassified = (DoubleToSignal(oPrevSignal) == Buy || DoubleToSignal(oPrevSignal) == Sell || DoubleToSignal(oPrevSignal) == Neutral);
if(oClassified)
{
m_oosSamples++;
m_oosConfidenceSum += MathAbs(oPrevSignal);
if(dOosError < 0)
dOosError = 0;
bool hit = (DoubleToSignal(oPrevSignal) == oTrueSignal);
// Per-class confusion counts, used for the Buy/Sell recall convergence gate below
switch(oTrueSignal)
{
case Buy:
m_oosBuyTotal++;
if(hit)
m_oosBuyHits++;
break;
case Sell:
m_oosSellTotal++;
if(hit)
m_oosSellHits++;
break;
default:
m_oosNeutralTotal++;
if(hit)
m_oosNeutralHits++;
break;
}
// Same confusion counts keyed by what the model actually PREDICTED this bar, not the
// true label - see m_oosBuyPredicted's declaration comment for why recall alone can
// hide an over-firing class.
switch(DoubleToSignal(oPrevSignal))
{
case Buy:
m_oosBuyPredicted++;
if(hit)
m_oosBuyPredictedHits++;
break;
case Sell:
m_oosSellPredicted++;
if(hit)
m_oosSellPredictedHits++;
break;
default:
m_oosNeutralPredicted++;
if(hit)
m_oosNeutralPredictedHits++;
break;
}
if(hit)
{
dOosForecast += (100 - dOosForecast) / Net.recentAverageSmoothingFactor;
dOosError -= dOosError / Net.recentAverageSmoothingFactor;
}
else
{
dOosForecast -= dOosForecast / Net.recentAverageSmoothingFactor;
dOosError += (100 - dOosError) / Net.recentAverageSmoothingFactor;
}
}
// Mirror pass 1's chart annotation for this (OOS) bar, now using post-training weights
// instead of pass 1's pre-training snapshot - last write wins on the shared per-bar-time
// object, so this supersedes pass 1's earlier draw for the same bar with the correct one.
m_lastBarTime = m_Time.GetData(oi);
if(oi > 0)
{
if(DoubleToSignal(oPrevSignal) == Neutral)
DeleteObject(m_lastBarTime);
else
DrawObject(m_lastBarTime, oPrevSignal, m_High.GetData(oi), m_Low.GetData(oi));
}
}
if(m_oosScoreIndex - 1 >= 2 && GetTickCount() - chunkStartTick >= TRAIN_TIME_BUDGET_MS)
{
//--- yield: save enough to resume PASS 3 mid-walk on the next call - m_isPass3Active and
//--- m_oosScoreIndex (both members) carry the actual resume position.
m_resumeBars = bars;
m_resumeTotalIter = totalIter;
m_resumeOosCutoff = oosCutoff;
m_resumeAddLoop = add_loop;
m_resumeBarIndex = i;
m_eraResumePending = true;
m_modelEta = eta;
return;
}
}
m_isPass3Active = false;
}
//--- Diagnostic recall snapshot for the periodic progress log further below - populated inside
//--- the m_oosSamples>0 recall-gate block when this era actually computes it; stays -1 ("n/a"
//--- in the log) on eras that don't (era 0, or a stopped/cap-hit era).
int logBuyRecallPct = -1, logSellRecallPct = -1, logNeutralRecallPct = -1;
//--- Predicted-rate (of all OOS bars this era, how often the model called this class at all) and
//--- precision (of the calls it made, how many were right) for Buy/Sell - m_oosBuyPredicted/
//--- m_oosSellPredicted (see that member's declaration comment) were already being tracked for
//--- exactly this but never surfaced anywhere. A recall-only view can't tell "the model never once
//--- calls Sell" (predicted rate stuck at 0%) apart from "the model calls Sell plenty but always on
//--- the wrong bars" (predicted rate healthy, precision near 0%) - both show up identically as 0%
//--- Sell recall, but point at completely different problems (a suppressed/dead output vs. a
//--- miscalibrated decision boundary), so this splits them out.
int logBuyPredPct = -1, logSellPredPct = -1, logBuyPrecPct = -1, logSellPrecPct = -1;
bool shouldLogProgress = false;
//--- era complete (ran out of bars) or a stop was requested mid-era
if(add_loop)
{
m_eraCount++;
m_erasSinceCooldown++;
//--- EMA shadow-weight deployment: blend the shadow a small step (SHADOW_WEIGHT_TAU) toward
//--- Net's just-updated weights, every era - see m_shadowNet's declaration comment. Must run
//--- here, inside the era loop, not just once at Train()-end: the whole point is damping the
//--- WITHIN-run oscillation (era-to-era whipsaw), which a single end-of-run blend would miss
//--- entirely.
EnsureShadowNet();
if(CheckPointer(m_shadowNet) != POINTER_INVALID)
m_shadowNet.BlendWeightsFrom(Net, SHADOW_WEIGHT_TAU);
//--- Status-label progress is invisible with no chart (headless/optimization runs), and even in
//--- visual mode a long training run can otherwise look "stuck" for a long time with no
//--- Journal output at all - log progress at most every ~5s (real wall-clock, not simulated
//--- time) so an operator can tell it's actively working, not hung. The actual Print() is
//--- deferred past the recall-gate block below (see logBuyRecallPct etc.) so this line can
//--- show per-class OOS recall - once OOS accuracy alone clears the target, recall is the
//--- most common thing still silently blocking convergence, and previously had no visibility
//--- outside of a regression event.
static uint lastProgressLogTick = 0;
uint nowTick = GetTickCount();
shouldLogProgress = (nowTick - lastProgressLogTick >= 5000);
if(shouldLogProgress)
lastProgressLogTick = nowTick;
if(m_maxErasPerRun > 0 && m_erasSinceCooldown >= m_maxErasPerRun)
{
stop = true;
m_trainCooldownUntil = TimeCurrent() + m_trainRetryCooldownSeconds;
Print(ID + ": WARNING - hit the " + IntegerToString(m_maxErasPerRun) + "-era safety cap for this training run without converging (best OOS so far " + DoubleToString(dOosForecast, 1) + "%, target " + IntegerToString(m_stopTrainWR) + "%). Pausing for " + IntegerToString(m_trainRetryCooldownSeconds) + "s before trying again - the convergence targets (MinRecall / StopTrainWR / SwingConfirmationBars) may be unreachable for this config; consider relaxing them if this keeps happening.");
}
}
if(!stop)
{
dError = Net.getRecentAverageError();
if(add_loop)
{
if(m_oosSamples > 0)
{
// Confidence calibration (classification head only - see m_confidenceCalScale's
// declaration comment): compare this era's actual OOS accuracy against the average
// confidence magnitude the model claimed, EMA-blend the resulting scale into
// m_confidenceCalScale so SignedAIConfidence() reports something closer to a real
// probability instead of the raw, uncalibrated softmax value.
if(m_outputNeuronsCount == 3 && m_oosConfidenceSum > 0.0)
{
double empiricalAccuracy = (double)(m_oosBuyHits + m_oosSellHits + m_oosNeutralHits) / m_oosSamples;
double avgClaimedConfidence = m_oosConfidenceSum / m_oosSamples;
double eraScale = MathMax(0.3, MathMin(1.5, empiricalAccuracy / avgClaimedConfidence));
m_confidenceCalScale += (eraScale - m_confidenceCalScale) / Net.recentAverageSmoothingFactor;
}
// Per-class recall gate, symmetric across all three classes: a model that "wins" on
// blended dOosForecast purely by calling everything Neutral (or, just as biased, by
// over-calling Buy/Sell at Neutral's expense) would still pass a plain accuracy check -
// require Buy, Sell, AND Neutral OOS recall to each individually clear
// m_minDirectionalRecallPct so the network can't converge while biased toward any one
// output. A class with FEWER than MIN_OOS_CLASS_SAMPLES_FOR_GATE true OOS samples this
// era doesn't block (recallPct == -1 => treated as passing) so a thin OOS window doesn't
// deadlock convergence early in a run. Computed BEFORE the checkpoint/eta-decay block
// below (not just the final m_objectiveMet gate) so "best" ranking is recall-aware too -
// see isBetterEra's comment for why that matters.
//
// The threshold matters: a bare ">0" here (the original behavior) let a run converge at
// era 44-46 with the OOS window containing exactly ZERO true Buy/Sell bars that era
// (logged as "OOS recall Buy:n/a Sell:n/a Neutral:100%") - a full Neutral-only collapse
// that the gate waved through because there was nothing to measure recall against, not
// because the model was actually unbiased. Requiring a real minimum sample count means
// an unlucky/thin OOS slice blocks convergence instead of silently passing it.
int buyRecallPct = (m_oosBuyTotal >= MIN_OOS_CLASS_SAMPLES_FOR_GATE) ? (int)MathRound(100.0 * m_oosBuyHits / m_oosBuyTotal) : -1;
int sellRecallPct = (m_oosSellTotal >= MIN_OOS_CLASS_SAMPLES_FOR_GATE) ? (int)MathRound(100.0 * m_oosSellHits / m_oosSellTotal) : -1;
int neutralRecallPct = (m_oosNeutralTotal >= MIN_OOS_CLASS_SAMPLES_FOR_GATE) ? (int)MathRound(100.0 * m_oosNeutralHits / m_oosNeutralTotal) : -1;
logBuyRecallPct = buyRecallPct;
logSellRecallPct = sellRecallPct;
logNeutralRecallPct = neutralRecallPct;
// Predicted-rate (share of ALL OOS bars this era the model called this class, regardless
// of whether that call was right) and precision (of just those calls, how many were
// right) - see logBuyPredPct's declaration comment above for why this is worth logging
// alongside recall. Gated on m_oosSamples>0 (already true inside this block) rather than
// MIN_OOS_CLASS_SAMPLES_FOR_GATE - unlike recall, the denominator here is every OOS bar,
// not just this class's true bars, so it stays meaningful even in an era where the model
// never once calls this class (predicted rate legitimately 0%, not a thin-sample n/a).
logBuyPredPct = (int)MathRound(100.0 * m_oosBuyPredicted / m_oosSamples);
logSellPredPct = (int)MathRound(100.0 * m_oosSellPredicted / m_oosSamples);
logBuyPrecPct = (m_oosBuyPredicted > 0) ? (int)MathRound(100.0 * m_oosBuyPredictedHits / m_oosBuyPredicted) : -1;
logSellPrecPct = (m_oosSellPredicted > 0) ? (int)MathRound(100.0 * m_oosSellPredictedHits / m_oosSellPredicted) : -1;
bool directionalRecallOK = (buyRecallPct < 0 || buyRecallPct >= m_minDirectionalRecallPct) &&
(sellRecallPct < 0 || sellRecallPct >= m_minDirectionalRecallPct) &&
(neutralRecallPct < 0 || neutralRecallPct >= m_minDirectionalRecallPct);
// A real (non-thin-sample, i.e. not the -1 "n/a" sentinel) 0% recall on any class means
// the model never once got that class right this era - a majority-class collapse
// (predict-everything-Neutral, or symmetrically a Buy/Sell-only collapse), not progress
// toward separating classes. Before any era has ever passed the recall floor,
// isBetterEra's fallback below is a pure blended-accuracy tiebreak, and blended accuracy
// is trivially maximized by collapsing to the majority class. Observed in practice
// (2026-07-19, SP500 H4): once a run landed on a 0%/0%/100% Buy/Sell/Neutral era, its
// accuracy kept creeping upward for 124 STRAIGHT eras purely from sharpening the
// Neutral-vs-everything boundary - each tick registered as a "new best", re-anchoring the
// checkpoint AND bumping eta back toward its ceiling (the recovery bump below), actively
// rewarding the collapse instead of remaining neutral to it. Excluding these eras from
// isBetterEra denies them that anchor/reward without touching the restore/decay branch
// below, which stays exactly as gated on m_bestPassedRecall as before - see that block's
// own comment for why loosening THAT part pre-pass caused a worse failure historically.
bool isFullyCollapsedEra = (buyRecallPct == 0 || sellRecallPct == 0 || neutralRecallPct == 0);
// Lexicographic "better than the best-so-far" ordering: passing the directional recall
// floor always outranks not passing it, regardless of blended dOosForecast; only WITHIN
// the same pass/fail category does blended accuracy break the tie. Without this, an era
// that traded a few "safe" Neutral calls for genuinely useful (recall-improving) Buy/Sell
// calls would look like a regression in blended-accuracy-only terms and get its
// checkpoint skipped / learning rate cut - fighting directly against the network learning
// to call Buy/Sell at all, since Neutral is the large majority class (~80%+ of labels) and
// a model that just calls everything Neutral already scores well on blended accuracy
// alone. isWorseEra mirrors the same ordering for the eta-decay-on-regression trigger.
// (directionalRecallOK implies !isFullyCollapsedEra already, since the floor is always
// >0%, so the first clause below needs no extra guard - only the pre-pass accuracy-only
// tiebreak in the second clause does.)
bool isBetterEra = (directionalRecallOK && !m_bestPassedRecall) ||
(directionalRecallOK == m_bestPassedRecall && !isFullyCollapsedEra && dOosForecast > m_bestOosForecast);
// The recall-pass-loss clause used to fire on ANY drop out of a full 3-way recall pass,
// even a near-miss on one class at unchanged accuracy (e.g. observed: Buy:56% Sell:41%
// Neutral:34% - Neutral alone missing the 40% floor by a few points) - treating that
// identically to a total collapse back to Neutral-only. With three classes all needing
// to simultaneously clear the floor, that made isWorseEra fire on most eras once a pass
// was ever achieved, ratcheting eta toward ETA_MIN within a handful of eras and then
// (before the recovery bump below existed) leaving it stuck there permanently - visible
// in practice as ~25 back-to-back identical "regressed from best 70.6% to 70.6%" eras.
// Now only counts as worse if accuracy ALSO dropped meaningfully (same threshold
// regardless of whether the recall-pass flag changed too) - losing the recall-pass flag
// at flat/improved accuracy is borderline variance, not a regression worth
// restoring+decaying over.
bool isWorseEra = dOosForecast < m_bestOosForecast - ETA_DECAY_REGRESSION_PCT;
if(isBetterEra)
{
m_bestOosForecast = dOosForecast;
m_bestPassedRecall = directionalRecallOK;
m_haveOosCheckpoint = Net.SaveCheckpoint(m_fileName + "_ckpt.tmp");
// Recovery bump: ETA_DECAY_FACTOR-only ever shrinks eta, and previously nothing ever
// grew it back - a losing streak early in a run (even a since-corrected one) would
// permanently cap how fast every later era could learn for the rest of the run, all
// the way down to ETA_MIN with no way back. A genuinely better era (new best, not
// just a tie) means the current eta is working, so nudge it back up a bit - capped at
// this model's own configured starting rate (m_etaCeiling - AdamLearningRate for
// ADAM, SgdLearningRate for SGD, see that member's declaration comment) so this
// can't runaway past the rate training was actually tuned to start at.
eta = MathMin(m_etaCeiling, eta / ETA_DECAY_FACTOR);
}
else
if(isWorseEra && m_bestOosForecast > 0)
{
// Decaying eta alone only softens FUTURE steps - it does nothing to undo the
// regression this era already baked into the weights, so a run could (and in
// practice did) spend 15+ eras compounding forward from one bad era's damage,
// each new era fighting the last one's overshoot instead of building on the best
// state found so far. Restore the last checkpointed-good weights before continuing
// (mirrors what FinalizeTrainRun() does at the END of a run, just applied live so
// the oscillation can't compound within a single run) - this is what actually turns
// "reduce LR on regression" into "step back, then retry slower", not just "drift
// slower".
//
// BOTH the restore AND the eta decay below are gated on m_bestPassedRecall: before
// ANY era has ever cleared the per-class recall floor, isBetterEra's own
// lexicographic ordering degrades to a pure blended-accuracy tiebreak
// (directionalRecallOK==false on both sides of the comparison), so "best checkpoint"
// during that phase just means "called Neutral most confidently so far" - restoring
// it would actively defend the majority-class collapse against any era that trades
// some accuracy for real Buy/Sell recall, which is exactly the bias this whole
// recall-gate mechanism exists to prevent (see isBetterEra's own comment above).
// Observed in practice: era 1-3 all "improved" on accuracy alone
// (24.9%->41.4%->52.3%) while Buy/Sell recall stayed at a flat 0% the entire time -
// restoring pre-pass would have locked training into that trajectory instead of
// letting it explore past it. Decaying eta has the same bias one step removed:
// every regression relative to a Neutral-collapse "best" shrinks eta a little more,
// steadily strangling the exploration needed to escape that collapse until eta
// bottoms out at ETA_MIN with no real solution ever found and no checkpoint to fall
// back on either - observed in practice as a run whose best-ever blended accuracy
// kept landing on 0%/0%/100% Buy/Sell/Neutral recall eras, each one triggering
// another decay on the very next era, until eta floored out around era 20 and the
// remaining eras just oscillated between collapse states with no way to make a
// large-enough move to escape and no way to reset. Once m_bestPassedRecall is true,
// there IS a genuinely good state worth protecting, and both restoring the
// checkpoint and decaying eta on regression are safe/correct again.
if(m_bestPassedRecall)
{
if(m_haveOosCheckpoint && Net.LoadCheckpoint(m_fileName + "_ckpt.tmp"))
dOosForecast = m_bestOosForecast;
if(eta > ETA_MIN)
eta = MathMax(ETA_MIN, eta * ETA_DECAY_FACTOR);
Print(ID + ": OOS accuracy regressed from best " + DoubleToString(m_bestOosForecast, 1) +
"% to " + DoubleToString(dOosForecast, 1) + "% - restoring best checkpoint and decaying learning rate to " + DoubleToString(eta, 6));
}
else
Print(ID + ": OOS accuracy regressed from best " + DoubleToString(m_bestOosForecast, 1) +
"% to " + DoubleToString(dOosForecast, 1) + "% - no recall-passing checkpoint yet, letting training continue past this without decaying eta (still " + DoubleToString(eta, 6) + ")");
}
m_oosWindow.Add(dOosForecast);
while(m_oosWindow.Total() > STABILITY_WINDOW)
m_oosWindow.Delete(0);
m_oosStable = false;
if(m_oosWindow.Total() >= STABILITY_WINDOW)
{
double oosMin = m_oosWindow.At(0), oosMax = m_oosWindow.At(0);
for(int w = 1; w < m_oosWindow.Total(); w++)
{
oosMin = MathMin(oosMin, m_oosWindow.At(w));
oosMax = MathMax(oosMax, m_oosWindow.At(w));
}
m_oosStable = (oosMax - oosMin) <= STABILITY_TOLERANCE;
}
// The dError<0.1 RMS-error floor is meaningful for the single-neuron regression head
// (m_outputNeuronsCount==1), where it's the only convergence signal available. For the
// 3-neuron one-hot classification head it's redundant with, and far stricter than,
// dOosForecast/directionalRecallOK: reaching RMS error 0.1 across 3 one-hot targets
// needs every output neuron within ~0.17 of its target on average, i.e. near-perfect
// confident calibration on EVERY bar, not just correct argmax calls - unreachable in
// practice under normal market label noise, so classification runs would oscillate
// forever (era after era hitting good OOS accuracy and passing recall, but never
// satisfying this) without this carve-out.
bool errorGateOK = (m_outputNeuronsCount == 3) ? true : (dError < 0.1);
m_objectiveMet = (dOosForecast >= m_stopTrainWR) && errorGateOK && directionalRecallOK;
}
// Only mark the persisted model "complete" once it actually converged this era -
// an interruption (stop) or an ordinary in-progress era must stay flagged incomplete
// so a restart resumes training instead of quietly treating a partial run as done.
m_trainingComplete = (m_objectiveMet && m_oosStable);
double currentIndicatorParams[];
m_indicatorTuner.Flatten(currentIndicatorParams);
Net.Save(m_activeFileName + ".nnw", dError, dUndefine, dForecast, dtStudied, m_activeFileCommon, m_eraCount, m_trainingComplete, currentIndicatorParams);
SaveShadowNet(currentIndicatorParams);
}
}
if(shouldLogProgress)
{
string recallInfo = (logBuyRecallPct < 0 && logSellRecallPct < 0 && logNeutralRecallPct < 0) ? "" :
(" | OOS recall Buy:" + (logBuyRecallPct < 0 ? "n/a" : IntegerToString(logBuyRecallPct) + "%") +
" Sell:" + (logSellRecallPct < 0 ? "n/a" : IntegerToString(logSellRecallPct) + "%") +
" Neutral:" + (logNeutralRecallPct < 0 ? "n/a" : IntegerToString(logNeutralRecallPct) + "%") +
" (need >=" + IntegerToString(m_minDirectionalRecallPct) + "% each)");
// See logBuyPredPct's declaration comment for why this is worth logging alongside recall -
// it's what tells apart a suppressed/dead output (predicted rate stuck at 0%) from a
// miscalibrated boundary (predicted rate healthy, precision poor), which look identical from
// recall alone.
string predictedInfo = (logBuyPredPct < 0 && logSellPredPct < 0) ? "" :
(" | OOS calls Buy:" + (logBuyPredPct < 0 ? "n/a" : IntegerToString(logBuyPredPct) + "%") +
" (prec " + (logBuyPrecPct < 0 ? "n/a" : IntegerToString(logBuyPrecPct) + "%") + ")" +
" Sell:" + (logSellPredPct < 0 ? "n/a" : IntegerToString(logSellPredPct) + "%") +
" (prec " + (logSellPrecPct < 0 ? "n/a" : IntegerToString(logSellPrecPct) + "%") + ")");
Print(ID + ": training in progress - era " + IntegerToString(m_eraCount) + ", OOS accuracy " + DoubleToString(dOosForecast, 1) + "% (target " + IntegerToString(m_stopTrainWR) + "%), IS error " + DoubleToString(dError, 2) + recallInfo + predictedInfo);
// Forced (unthrottled) panel refresh, right here alongside the console line above, using this
// era's own just-finalized m_eraCount/dOosForecast - see UpdateTrainingStatusLabel's
// declaration comment for why this can't just rely on the next throttled bar-scan call to
// catch up (it would, but a full era later than the console already reported it).
UpdateTrainingStatusLabel("Era complete", m_lastDisplayNeuron0, m_lastDisplayNeuron1, m_lastDisplayNeuron2, m_lastDisplaySignal, true);
}
//--- Genuine convergence THIS era (not a stale m_trainingComplete carried over from a previous
//--- run) - (re)start the evaluation-only continual-learning OOS walk. Always rebuilt fresh from
//--- the just-converged weights; never resumes a stale walk from a superseded model.
if(!stop && m_objectiveMet && m_oosStable)
{
Print(ID + ": training CONVERGED at era " + IntegerToString(m_eraCount) + " - OOS accuracy " + DoubleToString(dOosForecast, 1) +
"% (target " + IntegerToString(m_stopTrainWR) + "%), IS error " + DoubleToString(dError, 2) + " - stable for " +
IntegerToString((int)m_oosWindow.Total()) + " consecutive eras. Weights saved, switching to live inference.");
StartOosContinualSimulation(bars, oosCutoff);
}
if(stop || (m_objectiveMet && m_oosStable))
FinalizeTrainRun();
//--- else: this era is done but the run continues - the next Train() call (re-triggered via
//--- ScheduleTrainingIfNeeded()'s custom event, same mechanism as always) starts the next era
//--- fresh, since m_eraResumePending is false while m_trainRunActive stays true
//--- Save this model's own learning-rate trajectory back out of the shared global before
//--- returning - see m_modelEta's declaration comment. Covers every path that reaches here
//--- (natural era completion, whether or not the run itself just finalized).
m_modelEta = eta;
}
//+------------------------------------------------------------------+
//| Ends the current Train() run: restores the best-scoring era's |
//| checkpointed weights (if any beat the era the loop happened to |
//| end on), persists final state, and clears the resumable-run |
//| flags. Called both from Train() itself (natural stop/converge) |
//| and from StopTraining() (a mid-chunk Stop click won't get |
//| another "New Bar" event to resume into, since |
//| ScheduleTrainingIfNeeded() refuses to schedule while |
//| m_trainingStopRequested is set, so it must finalize synchronously |
//| there instead of being left dangling). |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::FinalizeTrainRun(void)
{
string checkpointFile = m_fileName + "_ckpt.tmp";
//--- deploy the most stable/best-scoring era's weights rather than whatever the run happened to
//--- end on (which may reflect drift after the objective was first hit, or an aborted run)
if(m_haveOosCheckpoint)
{
if(Net.LoadCheckpoint(checkpointFile))
{
dOosForecast = m_bestOosForecast;
RefreshLatestSignal();
double currentIndicatorParams[];
m_indicatorTuner.Flatten(currentIndicatorParams);
Net.Save(m_activeFileName + ".nnw", dError, dUndefine, dForecast, dtStudied, m_activeFileCommon, m_eraCount, m_trainingComplete, currentIndicatorParams);
SaveShadowNet(currentIndicatorParams);
}
FileDelete(checkpointFile);
}
if(m_eraCount > 0)
dtStudied = m_lastBarTime;
m_trainRunActive = false;
m_eraResumePending = false;
m_haveOosCheckpoint = false;
}
//+------------------------------------------------------------------+
//| (Re)sizes the label AND feature caches and clears them if `bars` |
//| (or the now-relative index frame) has changed since the last |
//| build - see the member declaration comments for why this is the |
//| correct invalidation trigger. Returns true if a rebuild happened. |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::EnsureBarCachesCapacity(int bars)
{
if(bars == m_labelCacheBars && m_Time.GetData(0) == m_labelCacheAnchorTime)
return false;
ArrayResize(m_labelCacheBuy, bars);
ArrayResize(m_labelCacheSell, bars);
ArrayResize(m_labelCacheHasValue, bars);
ArrayInitialize(m_labelCacheHasValue, false);
ArrayResize(m_featureCache, bars * m_neuronsCount);
ArrayResize(m_featureCacheHasValue, bars);
ArrayResize(m_featureCacheValid, bars);
ArrayInitialize(m_featureCacheHasValue, false);
m_labelCacheBars = bars;
m_labelCacheAnchorTime = m_Time.GetData(0);
return true;
}
//+------------------------------------------------------------------+
//| Lazy cache-miss fallback for a bar the eager prebuild pass (see |
//| AdvanceZigZagLabelState()) didn't cover - e.g. a new candle that |
//| closed after prebuild already completed. A genuine ZigZag pivot |
//| can only be confirmed once price has deviated enough FROM it on a |
//| LATER bar (see AdvanceZigZagLabelState()), which for a bar this |
//| close to "now" may not have happened yet - exactly like a live |
//| ZigZag indicator's most recent leg is always provisional/ |
//| repainting until confirmed. Rather than guess, this always labels |
//| Neutral; the sequential prebuild scan is what actually assigns |
//| Buy/Sell once there's enough subsequent price action to know. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::ComputeLabelForBar(int i, int bars, bool &buy, bool &sell)
{
buy = false;
sell = false;
}
//+------------------------------------------------------------------+
//| Confirms (or Neutral-labels) whichever candidate bar is exactly |
//| m_swingConfirmationBars bars behind the one currently being |
//| visited, by reading the real ADZigZag indicator's own verdict on |
//| it directly - see m_swingConfirmationBars' declaration comment |
//| for why that delay exists (a ZigZag's recent legs can repaint). |
//| Buy = candidate bar is a confirmed bottom (ADZigZagBuffer == its |
//| own Low), Sell = confirmed top (== its own High), else Neutral - |
//| this is the "will the next candle be the ZigZag reversal" label |
//| itself, at whatever now-relative index the feature window ends |
//| at (unchanged from the old fractal-based labeling's index |
//| convention - see BufferTempData()'s callers for how a training |
//| example's feature window and its label index line up). |
//| MUST be called in strict oldest-to-newest order (decreasing |
//| now-relative index i) - AdvanceLabelCachePrebuild() (the only |
//| caller) already guarantees this, though unlike the old fractal |
//| scan this replaced, no state actually carries between calls here. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::AdvanceZigZagLabelState(int i, int bars)
{
int idx = i + MathMax(m_swingConfirmationBars, 1);
if(idx >= bars || m_labelCacheHasValue[idx])
return;
bool buy = false, sell = false;
double zz = m_ADZigZag.GetData(0, idx);
if(zz != 0.0)
{
double hi = m_High.GetData(idx);
double lo = m_Low.GetData(idx);
if(zz >= hi - _Point)
sell = true; // confirmed top -> Sell
else if(zz <= lo + _Point)
buy = true; // confirmed bottom -> Buy
}
m_labelCacheBuy[idx] = buy;
m_labelCacheSell[idx] = sell;
m_labelCacheHasValue[idx] = true;
}
//+------------------------------------------------------------------+
//| Nearest confirmed ZigZag pivot at fromIdx or older (now-relative |
//| index, so "older" means scanning with INCREASING p - see this |
//| file's now-relative-index convention, same as AdvanceZigZagLabel- |
//| State() above). Capped at SWING_SCAN_CAP_BARS so a long quiet |
//| stretch with no qualifying pivot can't turn this into an unbounded |
//| scan; returns false (no pivot found) rather than looping forever |
//| if the cap is hit or history runs out first. |
//| |
//| Caller's responsibility, not this method's: applying the |
//| m_swingConfirmationBars repainting embargo to fromIdx before |
//| calling. This method itself just finds the nearest nonzero |
//| ADZigZag buffer entry at/after whatever index it's given - it has |
//| no opinion on whether that index is safe to read yet. The ONE |
//| caller that needs the embargo (BufferTempDataCompute()'s |
//| m_useSwingContext block, looking up "the pivot as of THIS bar") |
//| applies it before the first call; the second call in that same |
//| block (finding the PRIOR completed leg, starting from pivotIdx+1) |
//| doesn't need to re-apply it - anything at or before an already- |
//| confirmed pivot is necessarily even older, hence already confirmed |
//| too. |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::FindConfirmedZigZagPivot(int fromIdx, int &pivotIdx, double &pivotPrice, bool &pivotIsLow)
{
for(int p = MathMax(fromIdx, 0); p < fromIdx + SWING_SCAN_CAP_BARS; p++)
{
if(m_Open.GetData(p) == EMPTY_VALUE)
return false; // ran off the end of available history
double zz = m_ADZigZag.GetData(0, p);
if(zz == 0.0)
continue;
pivotIdx = p;
pivotPrice = zz;
pivotIsLow = (zz <= m_Low.GetData(p) + _Point);
return true;
}
return false;
}
//+------------------------------------------------------------------+
//| Kicks off the one-time eager label-cache pre-build for a fresh |
//| start (see m_labelCachePrebuilt's declaration comment). Computes |
//| the bar count/OOS split exactly as Train()'s era-start block |
//| would, then arms AdvanceLabelCachePrebuild() to do the actual |
//| chunked scan on this and subsequent Train() calls. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::StartLabelCachePrebuild(void)
{
int barsNow = (int)MathMin(Bars(m_symbol.Name(), PERIOD_CURRENT, dtStudied, TimeCurrent()) + m_historyBars, Bars(m_symbol.Name(), PERIOD_CURRENT));
if(!ResizeBuffers(barsNow) || !RefreshData())
return; // couldn't prep buffers yet - m_labelCachePrebuilt stays false, retried next call
EnsureBarCachesCapacity(barsNow);
int totalIter = (int)MathMax(barsNow - MathMax(m_historyBars, 0), 0);
m_labelPrebuildBars = barsNow;
m_labelPrebuildOosCutoff = (int)(MathMax(0, MathMin(100, m_oosSplitPct)) / 100.0 * totalIter);
m_labelPrebuildIndex = (int)(barsNow - MathMax(m_historyBars, 0) - 1);
m_labelPrebuildBuyCount = 0;
m_labelPrebuildSellCount = 0;
m_labelPrebuildNeutralCount = 0;
m_labelPrebuildActive = true;
}
//+------------------------------------------------------------------+
//| Advances the eager label-cache pre-build by up to a time budget, |
//| then yields (same chunking pattern as the era loop). Mirrors the |
//| era loop's own labeling eligibility gate (minus the dPrevSignal |
//| check, meaningless pre-first-feedForward). On completion, seeds |
//| m_prevEraTrueBuyCount/Sell/Neutral from the upfront IS-only tally |
//| so era 0 gets real class-balance oversampling instead of reps=1. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::AdvanceLabelCachePrebuild(void)
{
const uint PREBUILD_TIME_BUDGET_MS = 80;
uint chunkStartTick = GetTickCount();
int i;
for(i = m_labelPrebuildIndex; i >= 2; i--)
{
if(GetTickCount() - chunkStartTick >= PREBUILD_TIME_BUDGET_MS)
{
m_labelPrebuildIndex = i;
return;
}
if(!(i < (int)(m_labelPrebuildBars - MathMax(m_historyBars, 0) - 1) && m_Time.GetData(i) > dtStudied))
continue;
// Unlike the old per-bar-independent ComputeLabelForBar(), a proper ZigZag pivot can only be
// confirmed by a LATER (more recent, lower-index) bar than the candidate itself - see
// AdvanceZigZagLabelState()'s declaration comment. So a bar's final Buy/Sell/Neutral label is
// NOT necessarily known the moment we visit it here; it may get retroactively overwritten by a
// later iteration of this same loop. The IS/OOS Buy/Sell/Neutral tally below is therefore done
// in a separate final pass AFTER the whole scan completes, not incrementally per-bar like the
// old cache-hit-or-compute code here used to.
if(!m_labelCacheHasValue[i])
AdvanceZigZagLabelState(i, m_labelPrebuildBars);
}
//--- Sequential scan complete - every bar's own exact-pivot label is final. Widen each confirmed
//--- pivot into its +-LABEL_WINDOW_BARS neighborhood (see that constant's declaration comment)
//--- before the tally pass below, so both the logged "IS true-label distribution" and era 0's
//--- oversampling ratio reflect what the network is actually trained/scored against. Spreads from a
//--- frozen SNAPSHOT of the just-resolved genuine pivots, never from the live m_labelCache* arrays
//--- being written below - reading live here would let a just-written window cell (i-1, i-2, ...)
//--- get visited later in this same decreasing-i pass, look indistinguishable from a genuine pivot,
//--- and re-spread from itself - a single real pivot cascading across the entire array instead of
//--- staying bounded to +-LABEL_WINDOW_BARS (observed in practice: one training run's IS tally came
//--- back Buy 5000 / Sell 4657 / Neutral 8 out of ~9665 bars).
if(LABEL_WINDOW_BARS > 0)
{
bool origBuy[], origSell[];
ArrayCopy(origBuy, m_labelCacheBuy);
ArrayCopy(origSell, m_labelCacheSell);
int maxIdx = m_labelPrebuildBars - 1;
for(i = maxIdx; i >= 2; i--)
{
if(!origBuy[i] && !origSell[i])
continue; // not a genuine confirmed pivot in the original scan - nothing to spread from
bool isBuy = origBuy[i];
for(int w = 1; w <= LABEL_WINDOW_BARS; w++)
{
int lo = i - w, hi = i + w;
// Only claim a neighbor that wasn't ITSELF a genuine pivot in the ORIGINAL snapshot - a
// real pivot (of either class) always keeps its own exact label; overlapping window-bleed
// from two nearby pivots may freely overwrite each other (harmless - neither is ground truth).
if(lo >= 2 && !origBuy[lo] && !origSell[lo])
{
m_labelCacheBuy[lo] = isBuy;
m_labelCacheSell[lo] = !isBuy;
m_labelCacheHasValue[lo] = true;
}
if(hi <= maxIdx && !origBuy[hi] && !origSell[hi])
{
m_labelCacheBuy[hi] = isBuy;
m_labelCacheSell[hi] = !isBuy;
m_labelCacheHasValue[hi] = true;
}
}
}
}
//--- Final tally pass (IS-only, matches isOOS = (i < oosCutoff) used by the era loop) to seed era
//--- 0's oversampling ratio - now over the widened labels, not just the exact pivot bars.
for(i = m_labelPrebuildBars - MathMax(m_historyBars, 0) - 1; i >= MathMax(2, m_labelPrebuildOosCutoff); i--)
{
if(!m_labelCacheHasValue[i])
continue; // e.g. bar was outside the dtStudied/window-edge eligibility gate above
if(m_labelCacheBuy[i])
m_labelPrebuildBuyCount++;
else
if(m_labelCacheSell[i])
m_labelPrebuildSellCount++;
else
m_labelPrebuildNeutralCount++;
}
//--- Prebuild complete - seed era 0's oversampling ratio from the real upfront tally instead of
//--- leaving it at the reps=1 fallback (see m_prevEraTrueBuyCount's declaration comment). Consumed
//--- (and cleared) by Train()'s era-start block on era 0 specifically - see m_prebuildSeedPending.
m_prevEraTrueBuyCount = m_labelPrebuildBuyCount;
m_prevEraTrueSellCount = m_labelPrebuildSellCount;
m_prevEraTrueNeutralCount = m_labelPrebuildNeutralCount;
m_prebuildSeedPending = true;
m_labelCachePrebuilt = true;
m_labelPrebuildActive = false;
Print(ID + ": label cache pre-built - IS true-label distribution -> Buy: " + IntegerToString(m_labelPrebuildBuyCount) +
" | Sell: " + IntegerToString(m_labelPrebuildSellCount) + " | Neutral: " + IntegerToString(m_labelPrebuildNeutralCount) +
" (seeding era 0's class-balance oversampling)");
//--- Cold-start fix: a freshly-initialized (random-weight) network's argmax is close to uniform
//--- noise across the 3 classes, so on this typically heavily-skewed label distribution it fires
//--- far more non-majority-class calls at the very start of era 0 than the true base rate warrants,
//--- until enough backProp steps correct it. Now that the real prior is known, push the output
//--- layer's bias toward whichever class actually dominates - only the bias term moves, the
//--- per-input weights stay randomly initialized and still carry the real learning signal. Only
//--- meaningful for the 3-output classification head, and only for a fresh net (this whole prebuild
//--- path is skipped entirely when a trained net was loaded from disk - see m_labelCachePrebuilt).
if(m_outputNeuronsCount == 3)
{
int dominant = 2; // Neutral
int dominantCount = m_labelPrebuildNeutralCount;
if(m_labelPrebuildBuyCount > dominantCount)
{
dominant = 0;
dominantCount = m_labelPrebuildBuyCount;
}
if(m_labelPrebuildSellCount > dominantCount)
{
dominant = 1;
dominantCount = m_labelPrebuildSellCount;
}
int totalLabeled = m_labelPrebuildBuyCount + m_labelPrebuildSellCount + m_labelPrebuildNeutralCount;
if(totalLabeled > 0 && (double)dominantCount / totalLabeled > 0.40)
{
const double BIAS_MAGNITUDE = 3.0; // sigmoid(+-3) ~= 0.95/0.05 - comfortably outweighs a
// fresh network's random per-input weighted-sum noise
double biasValues[3] = { -BIAS_MAGNITUDE, -BIAS_MAGNITUDE, -BIAS_MAGNITUDE };
biasValues[dominant] = BIAS_MAGNITUDE;
if(Net.SeedOutputLayerBias(biasValues))
PrintVerbose(ID + ": seeded output layer bias toward " + EnumToString((ENUM_SIGNAL)(dominant == 0 ? Buy : dominant == 1 ? Sell : Neutral)) +
" (era 0 cold-start fix)");
}
}
}
//+------------------------------------------------------------------+
//| Clones the just-converged Net into a separate CNet (m_simOosNet) |
//| and arms a chunked bar-by-bar walk through the OOS window - see |
//| AdvanceOosSimulationChunk(). Evaluation-only: the clone's learned |
//| weights are never written back to Net or any persisted file. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::StartOosContinualSimulation(int bars, int oosCutoff)
{
if(m_simOosRunActive)
{
delete m_simOosNet;
m_simOosNet = NULL;
m_simOosRunActive = false;
}
if(oosCutoff <= 0)
return; // nothing to walk this run
//--- Clone via the full Save()/Load() pair, NOT SaveCheckpoint()/LoadCheckpoint(): Load() calls
//--- InitOpenCL()/InitDirectML() before reconstructing layers, so a bare "new CNet(NULL)" ends up
//--- with a GPU/DirectML backend matching production. LoadCheckpoint() instead assumes opencl/
//--- directml are ALREADY initialized (it passes them straight into new CLayer(...)) - on a fresh
//--- CNet(NULL) (whose constructor no-ops for a NULL description) those are unset, which would
//--- silently build a wrong/degenerate clone.
string simFile = m_fileName + "_simoos.tmp";
double ip[];
if(!Net.Save(simFile, 0.0, 0.0, 0.0, dtStudied, false, m_eraCount, m_trainingComplete, ip))
return;
m_simOosNet = new CNet(NULL);
double loadE, loadU, loadF;
datetime loadTime;
long loadEra;
bool loadComplete;
double loadIp[];
bool loaded = m_simOosNet.Load(simFile, loadE, loadU, loadF, loadTime, false, loadEra, loadComplete, loadIp);
FileDelete(simFile);
if(!loaded)
{
delete m_simOosNet;
m_simOosNet = NULL;
return;
}
m_simOosCutoff = oosCutoff;
m_simOosBarIndex = oosCutoff - 1;
m_simOosForecast = 0;
m_simOosSamples = 0;
m_simOosRunActive = true;
}
//+------------------------------------------------------------------+
//| Advances the evaluation-only continual-learning OOS walk by up |
//| to TRAIN_TIME_BUDGET_MS of work, then yields (same chunking |
//| pattern as the real era loop's m_eraResumePending). For each bar, |
//| oldest-OOS to newest: predict with the clone's CURRENT weights, |
//| score against the cached true label, THEN let the clone learn |
//| from it (single pass, no oversampling replay) - simulating how |
//| the model would adapt bar-by-bar in real forward trading. Never |
//| touches Net, never Saves the clone - purely an evaluation metric.|
//+------------------------------------------------------------------+
void CExpertSignalAIBase::AdvanceOosSimulationChunk(void)
{
const uint SIM_TIME_BUDGET_MS = 80;
uint chunkStartTick = GetTickCount();
int i;
for(i = m_simOosBarIndex; i >= 0; i--)
{
if(GetTickCount() - chunkStartTick >= SIM_TIME_BUDGET_MS)
{
m_simOosBarIndex = i;
return;
}
if(i >= ArraySize(m_labelCacheHasValue) || !m_labelCacheHasValue[i])
continue; // no cached label for this bar (e.g. right at a window edge) - nothing to learn from
TempData.Clear();
TempData.Reserve((int)m_historyBars * m_neuronsCount);
//--- Window ends AT (includes) bar i - see Train()'s matching r declaration comment for why.
int r = i;
bool bufferOk = true;
for(int b = 0; b < (int)m_historyBars; b++)
{
if(!BufferTempData(r + b))
{
bufferOk = false;
break;
}
}
if(!bufferOk || TempData.Total() < (int)m_historyBars * m_neuronsCount)
continue;
m_simOosNet.feedForward(TempData);
m_simOosNet.getResults(TempData);
double simSignal = (m_outputNeuronsCount == 3) ? ApplyClassificationSoftmax() : TempData[0];
bool buy = m_labelCacheBuy[i];
bool sell = m_labelCacheSell[i];
ENUM_SIGNAL trueSignal = buy ? Buy : (sell ? Sell : Neutral);
bool hit = (DoubleToSignal(simSignal) == trueSignal);
m_simOosSamples++;
if(hit)
m_simOosForecast += (100 - m_simOosForecast) / Net.recentAverageSmoothingFactor;
else
m_simOosForecast -= m_simOosForecast / Net.recentAverageSmoothingFactor;
TempData.Clear();
if(m_outputNeuronsCount == 1)
TempData.Add(buy && !sell ? 1 : !buy && sell ? -1 : 0);
else
if(m_outputNeuronsCount == 3)
{
TempData.Add(buy ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW);
TempData.Add(sell ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW);
TempData.Add((!buy && !sell) ? LABEL_SMOOTH_HIGH : LABEL_SMOOTH_LOW);
}
m_simOosNet.backProp(TempData);
}
delete m_simOosNet;
m_simOosNet = NULL;
m_simOosRunActive = false;
Print(ID + ": continual-learning OOS simulation complete - " + IntegerToString(m_simOosSamples) + " samples, accuracy " + DoubleToString(m_simOosForecast, 1) + "%");
}
//| Rebuilds only the enabled AD* CiCustom handles in place, so a new |
//| trial's member-struct param values take effect. Re-Create()-ing |
//| the existing CiCustom object (rather than removing/re-adding it |
//| to indicators) avoids adding the same pointer into the CIndicators|
//| collection twice, which would risk it being deleted twice on |
//| teardown - MQL5's CIndicators has no documented single-item |
//| remove, and CiCustom.Create() already releases its old handle. |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::ReInitADIndicators(CIndicators *indicators)
{
bool result = true;
if(m_useADCumulativeDelta)
result = InitADCumulativeDelta(indicators, false) && result;
if(m_useADShorteningOfThrust)
result = InitADShorteningOfThrust(indicators, false) && result;
if(m_useADWyckoffEventStream)
result = InitADWyckoffEventStream(indicators, false) && result;
if(m_useADWyckoffFailedStructure)
result = InitADWyckoffFailedStructure(indicators, false) && result;
if(m_useADWyckoffSignificantBarInversion)
result = InitADWyckoffSignificantBarInversion(indicators, false) && result;
return result;
}
//+------------------------------------------------------------------+
//| Outer loop around Train(): when AutoTuneIndicators is disabled, |
//| or no AD indicator is enabled, this is a pure pass-through to |
//| Train() (identical behavior to before this feature existed). |
//| Otherwise it randomly perturbs AD indicator inputs across trials, |
//| keeping whichever trial's Train() converged with the best OOS |
//| score, and leaves the indicators/weights matching that trial. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::TuneIndicatorsAndTrain(datetime StartTrainBar = 0)
{
bool anyADEnabled = (m_useADCumulativeDelta || m_useADShorteningOfThrust || m_useADWyckoffEventStream ||
m_useADWyckoffFailedStructure || m_useADWyckoffSignificantBarInversion);
if(!m_autoTuneIndicators || !anyADEnabled)
{
Train(StartTrainBar);
return;
}
string bestCheckpointFile = m_fileName + "_tunebest_ckpt.tmp";
//--- Train() is itself chunked (see its declaration comment) and now yields well before a trial
//--- finishes, so this trial loop can no longer assume one Train() call = one complete trial.
//--- m_tuneTrialIndex < 0 means no multi-trial tuning run is in progress - start a fresh one.
if(m_tuneTrialIndex < 0)
{
m_tuneTrialIndex = 0;
m_tuneBestOosForecast = -1;
m_tuneLastTrialWasWin = true; // trial 0 uses the current/best-known values as-is
m_tuneHaveBestCheckpoint = false;
m_tuneStartTrainBar = StartTrainBar;
}
if(m_tuneTrialIndex >= m_indicatorTuneTrials)
{
// all trials done - deploy the winner and finish up
if(!m_tuneLastTrialWasWin)
{
ReInitADIndicators(m_indicatorsPtr);
// Undo the losing trial's weights too, so deployed weights match the deployed (best-known)
// indicator params rather than whatever the last (losing) trial trained against.
if(m_tuneHaveBestCheckpoint && Net.LoadCheckpoint(bestCheckpointFile))
{
dOosForecast = m_tuneBestOosForecast;
// The deployed weights now match the best (converged) trial, not the losing last trial -
// reflect that so a restart doesn't needlessly redo a full training run.
m_trainingComplete = true;
RefreshLatestSignal();
}
}
if(m_tuneHaveBestCheckpoint)
FileDelete(bestCheckpointFile);
// Re-persist so the .nnw's indicator-param block matches whatever ended up deployed above
// (the last Train() call inside the loop may have saved a losing trial's params instead)
double finalIndicatorParams[];
m_indicatorTuner.Flatten(finalIndicatorParams);
Net.Save(m_activeFileName + ".nnw", dError, dUndefine, dForecast, dtStudied, m_activeFileCommon, m_eraCount, m_trainingComplete, finalIndicatorParams);
// m_shadowNet (if bootstrapped at all here) only ever tracked the LAST trial's era-by-era
// blend history - possibly a losing trial that Net just got overwritten away from via
// LoadCheckpoint() above. Force a fresh re-clone from the just-deployed winning weights rather
// than let a stale, unrelated trial's shadow keep being blended toward whatever trial runs next.
if(CheckPointer(m_shadowNet) != POINTER_INVALID)
{
delete m_shadowNet;
m_shadowNet = NULL;
}
EnsureShadowNet();
SaveShadowNet(finalIndicatorParams);
m_tuneTrialIndex = -1; // ready for the next multi-trial tuning run, if any
return;
}
//--- (re-)start this trial's setup only if Train() isn't already mid-run for it - resuming a
//--- chunked trial must not re-perturb params or rebuild a fresh (untrained) topology
if(!m_trainRunActive)
{
if(m_tuneTrialIndex > 0)
{
// Perturb from the current best-known values, not cumulative drift from a losing trial
m_indicatorTuner.PerturbRandom(m_useADCumulativeDelta, m_useADShorteningOfThrust, m_useADWyckoffEventStream, m_useADWyckoffFailedStructure, m_useADWyckoffSignificantBarInversion);
}
ReInitADIndicators(m_indicatorsPtr);
// Fresh, untrained topology for a fair per-trial comparison (same layer shape every trial -
// only indicator inputs vary, which is what's being searched here)
BuildFreshTopology();
dError = -1;
dUndefine = 0;
dForecast = 0;
dOosForecast = 0;
m_oosSamples = 0;
dOosError = -1;
}
Train(m_tuneStartTrainBar);
if(m_trainRunActive)
return; // this trial's Train() run yielded mid-chunk - resume it on the next call
//--- this trial's Train() run has fully finished (converged, hit the era cap, or was stopped)
if(m_trainingComplete && dOosForecast > m_tuneBestOosForecast)
{
m_tuneBestOosForecast = dOosForecast;
m_indicatorTuner.SaveAsBest();
m_tuneLastTrialWasWin = true;
// Train()'s own checkpoint restore already put this trial's best-era weights into Net;
// snapshot them here too so a later losing trial can be undone at the weight level, not
// just the indicator-param level.
m_tuneHaveBestCheckpoint = Net.SaveCheckpoint(bestCheckpointFile);
}
else
{
// Losing candidate: restore best-known values so the next trial perturbs from the winner
m_indicatorTuner.RestoreBest();
m_tuneLastTrialWasWin = false;
}
m_tuneTrialIndex++;
}
//+------------------------------------------------------------------+
//| Post-convergence "new bar" handler - see ScheduleTrainingIfNeeded()|
//| for why this exists: once m_trainingComplete is true, a plain new |
//| bar must NOT re-enter Train()'s full era loop (which resets the |
//| best-checkpoint/eta-decay tracking and runs real Net.backProp() |
//| passes again, silently perturbing an already-converged model |
//| forever, once per bar, with no way to ever actually finish). This |
//| only refreshes the price/indicator buffers and re-runs inference |
//| for the newest bar so dPrevSignal/the chart arrow stay current - |
//| identical cost to what Train() does per-bar, minus every bit of |
//| training (label caching, oversampling, backProp, checkpointing). |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::RefreshConvergedSignal(void)
{
int barsNow = (int)MathMin(Bars(m_symbol.Name(), PERIOD_CURRENT, dtStudied, TimeCurrent()) + m_historyBars, Bars(m_symbol.Name(), PERIOD_CURRENT));
if(!ResizeBuffers(barsNow) || !RefreshData())
return;
RefreshLatestSignal();
dtStudied = m_Time.GetData(0);
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::SaveShadowNet(const double &indicatorParams[])
{
if(CheckPointer(m_shadowNet) == POINTER_INVALID)
return;
m_shadowNet.Save(m_activeFileName + "_shadow.nnw", dError, dUndefine, dForecast, dtStudied, m_activeFileCommon, m_eraCount, m_trainingComplete, indicatorParams);
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::EnsureShadowNet(void)
{
if(CheckPointer(m_shadowNet) != POINTER_INVALID)
return;
string shadowFile = m_activeFileName + "_shadow.nnw";
if(FileIsExist(shadowFile, m_activeFileCommon ? FILE_COMMON : 0))
{
CNet *loaded = new CNet(NULL);
if(CheckPointer(loaded) != POINTER_INVALID)
{
double loadE, loadU, loadF;
datetime loadTime;
long loadEra;
bool loadComplete;
double loadIp[];
if(loaded.Load(shadowFile, loadE, loadU, loadF, loadTime, m_activeFileCommon, loadEra, loadComplete, loadIp))
{
m_shadowNet = loaded;
return;
}
delete loaded;
}
}
//--- No compatible persisted shadow - bootstrap from Net's current weights. Clone via the full
//--- Save()/Load() pair, not SaveCheckpoint()/LoadCheckpoint() - see
//--- StartOosContinualSimulation()'s matching comment for why (LoadCheckpoint() assumes
//--- opencl/directml are already initialized on the target CNet, which a bare "new CNet(NULL)"
//--- doesn't have).
if(CheckPointer(Net) == POINTER_INVALID)
return;
string cloneFile = m_fileName + "_shadowclone.tmp";
double ip[];
if(!Net.Save(cloneFile, 0.0, 0.0, 0.0, dtStudied, false, m_eraCount, m_trainingComplete, ip))
return;
CNet *clone = new CNet(NULL);
if(CheckPointer(clone) == POINTER_INVALID)
{
FileDelete(cloneFile);
return;
}
double loadE, loadU, loadF;
datetime loadTime;
long loadEra;
bool loadComplete;
double loadIp[];
bool loaded = clone.Load(cloneFile, loadE, loadU, loadF, loadTime, false, loadEra, loadComplete, loadIp);
FileDelete(cloneFile);
if(!loaded)
{
delete clone;
return;
}
m_shadowNet = clone;
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::RefreshLatestSignal(void)
{
int i = 0;
TempData.Clear();
TempData.Reserve((int)m_historyBars * m_neuronsCount);
//--- Window ends AT (includes) bar i - see Train()'s matching r declaration comment for why: this
//--- must be the same window Train() learned from, or the deployed model is being queried on a task
//--- it was never trained for.
int r = i;
for(int b = 0; b < (int)m_historyBars; b++)
{
int bar_t = r + b;
if(!BufferTempData(bar_t))
return;
}
if(TempData.Total() < (int)m_historyBars * m_neuronsCount)
return;
//--- Live trading/inference reads from the EMA shadow net, not Net directly - see m_shadowNet's
//--- declaration comment. Falls back to Net if the shadow isn't bootstrapped yet (should only be
//--- momentarily, on a genuinely fresh start before EnsureShadowNet() has run).
EnsureShadowNet();
CNet *deployNet = (CheckPointer(m_shadowNet) != POINTER_INVALID) ? m_shadowNet : Net;
deployNet.feedForward(TempData);
deployNet.getResults(TempData);
if(m_outputNeuronsCount == 1)
dPrevSignal = TempData[0];
else
if(m_outputNeuronsCount == 3)
dPrevSignal = ApplyClassificationSoftmax();
datetime bt = m_Time.GetData(i);
if(DoubleToSignal(dPrevSignal) == Neutral)
DeleteObject(bt);
else
DrawObject(bt, dPrevSignal, m_High.GetData(i), m_Low.GetData(i));
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
double CExpertSignalAIBase::ApplyClassificationSoftmax(void)
{
double maxLogit = MathMax(TempData.At(0), MathMax(TempData.At(1), TempData.At(2)));
double sum = 0;
for(int res = 0; res < 3; res++)
{
double temp = exp(TempData.At(res) - maxLogit);
sum += temp;
TempData.Update(res, temp);
}
for(int res = 0; res < 3; res++)
TempData.Update(res, TempData.At(res) / sum);
double pBuy = TempData.At(0);
double pSell = TempData.At(1);
double pNeutral = TempData.At(2);
// TempData.Maximum(0,3) scans left-to-right and keeps the FIRST index on a tie, so any tie
// (including the degenerate all-equal 0.3333/0.3333/0.3333 case from a collapsed/untrained net)
// always resolved to Buy (index 0) - silently turning "the model has no idea" into a directional
// trade. Buy/Sell now only win with a strict majority over BOTH other classes; every tie,
// 2-way or 3-way, falls through to Neutral.
if(pBuy > pSell && pBuy > pNeutral)
return pBuy; // Buy signal
if(pSell > pBuy && pSell > pNeutral)
return -pSell; // Sell signal
return 0; // Neutral signal (also the fallback on any tie)
}
//+------------------------------------------------------------------+
//| Converts a double to ENUM_SIGNAL. |
//| 3-output (softmax classification) case: dPrevSignal's *sign* |
//| alone already encodes the argmax-selected class (+prob for Buy, |
//| -prob for Sell, exactly 0.0 for Neutral - see Train()/ |
//| RefreshLatestSignal()), so classification here is pure argmax: |
//| whichever class the network actually picked, full stop. No |
//| magnitude threshold is applied - confidence magnitude is a |
//| separate concern, already exposed via AIConfidence()/ |
//| SignedAIConfidence() (MathAbs(dPrevSignal)/dPrevSignal) for the |
//| signal engine's own confidence-weighted filters/lot sizing/SLTP, |
//| so this keeps "which class" and "how confident" decoupled. |
//| 1-output (tanh regression) case: unrelated network shape, keeps |
//| the original 0.50 magnitude cutoff as a genuine confidence gate. |
//+------------------------------------------------------------------+
ENUM_SIGNAL CExpertSignalAIBase::DoubleToSignal(double value)
{
value = NormalizeDouble(value, 2); // Round 'value' to two decimal places
if(value < -1.0 || value > 1.0)
return Undefine; // out of range, e.g. the -2 "not yet studied" sentinel
if(m_outputNeuronsCount == 3)
{
if(value > 0.0)
return Buy;
if(value < 0.0)
return Sell;
return Neutral;
}
if(value > 0.50)
return Buy;
if(value < -0.50)
return Sell;
return Neutral;
}
//+------------------------------------------------------------------+
//| See the declaration comment (class body) for why this exists. |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::UpdateTrainingStatusLabel(const string &progressLine, double neuron0, double neuron1, double neuron2, double signalValue, bool forceRefresh)
{
// Cache regardless of whether this particular call actually redraws below - see
// m_lastDisplayNeuron0's declaration comment for why the era-end forced refresh needs these.
m_lastDisplayNeuron0 = neuron0;
m_lastDisplayNeuron1 = neuron1;
m_lastDisplayNeuron2 = neuron2;
m_lastDisplaySignal = signalValue;
// Throttle: SetStatusLabel()'s ChartRedraw() is real work that gets slower as more chart objects
// accumulate over a long backtest - calling it every single bar across three passes (instead of
// pass 1 alone, the original frequency) is what actually stalled a run for 1.5+ hours without
// finishing era 0, not the shuffle itself. ~5 updates/sec is still visually live. forceRefresh
// bypasses this - see this method's declaration comment for why the era-end call needs to.
const uint STATUS_LABEL_THROTTLE_MS = 200;
uint nowTick = GetTickCount();
if(!forceRefresh && m_lastStatusLabelUpdateTick != 0 && nowTick - m_lastStatusLabelUpdateTick < STATUS_LABEL_THROTTLE_MS)
return;
m_lastStatusLabelUpdateTick = nowTick;
// Recall/precision-by-class and the continual-learning OOS sim are still tracked (used for the
// convergence gate elsewhere) but dropped from the on-chart status text - it made the panel too
// tall/wordy for a one-line-per-metric display; the counts below are enough at a glance.
string classLine = StringFormat(
"Predicted -> Buy: %d Sell: %d Neutral: %d\n" +
"Actual -> Buy: %d Sell: %d Neutral: %d",
m_countBuySignals, m_countSellSignals, m_countNeutralSignals,
m_trueBuyCount, m_trueSellCount, m_trueNeutralCount
);
string oosErrStr = (m_oosSamples > 0 ? DoubleToString(dOosError, 2) : "N/A");
string s;
if(m_outputNeuronsCount == 1)
s = StringFormat(
ID + " : Study -> Era %d\n" +
"%s\n" +
"IS %d%% Acc: %.2f%% MSE: %.2f AvgErr: %.2f\n" +
"OOS %d%% Acc: %.2f%% Mismatch: %s Samples: %d\n" +
"Signal Neuron: %.5f\n" +
"Forecast: %s -> %.2f\n" +
"%s",
m_eraCount, progressLine,
100 - m_oosSplitPct, dForecast,
dError,
Net.getRecentAverageError(),
m_oosSplitPct, dOosForecast, oosErrStr, m_oosSamples,
neuron0,
EnumToString(DoubleToSignal(signalValue)), signalValue,
classLine
);
else
if(m_outputNeuronsCount == 3)
s = StringFormat(
ID + " : Study -> Era %d\n" +
"%s\n" +
"IS %d%% Acc: %.2f%% MSE: %.2f AvgErr: %.2f\n" +
"OOS %d%% Acc: %.2f%% Mismatch: %s Samples: %d\n" +
"Buy: %.5f Sell: %.5f Neutral: %.5f\n" +
"Forecast: %s -> %.2f\n" +
"%s",
m_eraCount, progressLine,
100 - m_oosSplitPct, dForecast,
dError,
Net.getRecentAverageError(),
m_oosSplitPct, dOosForecast, oosErrStr, m_oosSamples,
neuron0, neuron1, neuron2,
EnumToString(DoubleToSignal(signalValue)), signalValue,
classLine
);
else
s = "Invalid neuron count!";
SetStatusLabel(s);
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::DrawObject(datetime time, double signal, double high, double low)
{
double price = 0;
int arrow = 0;
color clr = 0;
ENUM_ARROW_ANCHOR anch = ANCHOR_BOTTOM;
switch(DoubleToSignal(signal))
{
case Buy:
price = low;
arrow = 217;
clr = clrBlue;
anch = ANCHOR_TOP;
break;
case Sell:
price = high;
arrow = 218;
clr = clrRed;
anch = ANCHOR_BOTTOM;
break;
}
if(price == 0 || arrow == 0)
return;
string name = TimeToString(time);
if(ObjectFind(0, name) < 0)
{
ResetLastError();
if(!ObjectCreate(0, name, OBJ_ARROW, 0, time, 0))
{
printf("Error of creating object %d", GetLastError());
return;
}
}
ObjectSetDouble(0, name, OBJPROP_PRICE, price);
ObjectSetInteger(0, name, OBJPROP_ARROWCODE, arrow);
ObjectSetInteger(0, name, OBJPROP_COLOR, clr);
ObjectSetInteger(0, name, OBJPROP_ANCHOR, anch);
ObjectSetString(0, name, OBJPROP_TOOLTIP, EnumToString(DoubleToSignal(signal)) + " " + DoubleToString(signal, 5));
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::DeleteObject(datetime time)
{
string name = TimeToString(time);
if(ObjectFind(0, name) >= 0)
ObjectDelete(0, name);
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::ResizeBuffers(int barIndex)
{
if(!m_Open.BufferResize(barIndex) || !m_Close.BufferResize(barIndex) || !m_High.BufferResize(barIndex) || !m_Low.BufferResize(barIndex))
return false;
if(m_useVolumes)
{
if(!m_Volumes.BufferResize(barIndex))
return false;
}
if(m_useTime)
{
if(!m_Time.BufferResize(barIndex))
return false;
}
// Unconditional (not gated by m_useATR): the ATR-normalization in BufferTempData() reads
// m_ATR.Main() regardless of whether ATR is enabled as an explicit extra input feature -
// m_useATR only controls that feature-count opt-in (see InitIndicators()'s "already init in the
// base class" comment), not whether ATR data itself needs to be kept live.
if(!m_ATR.BufferResize(barIndex))
return false;
// Unconditional, same reasoning as m_ATR above - m_ADZigZag is the training-label source (see its
// declaration comment), not an opt-in feature, so it's never gated by an m_use* flag.
if(!m_ADZigZag.BufferResize(barIndex))
return false;
if(m_useADCumulativeDelta)
{
if(!m_ADCumulativeDelta.BufferResize(barIndex))
return false;
}
if(m_useADShorteningOfThrust)
{
if(!m_ADShorteningOfThrust.BufferResize(barIndex))
return false;
}
if(m_useADWyckoffEventStream)
{
if(!m_ADWyckoffEventStream.BufferResize(barIndex))
return false;
}
if(m_useADWyckoffFailedStructure)
{
if(!m_ADWyckoffFailedStructure.BufferResize(barIndex))
return false;
}
if(m_useADWyckoffSignificantBarInversion)
{
if(!m_ADWyckoffSignificantBarInversion.BufferResize(barIndex))
return false;
}
return true;
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::RefreshData()
{
// CSeries/CIndicator::Refresh() is void - there is no per-call success/failure signal to
// propagate here. The real data-validity check happens downstream, per value, in
// BufferTempDataCompute() (EMPTY_VALUE / atr<=0 guards) - this function's job is only to ask
// every buffer to refresh, unconditionally, before that per-value check runs.
m_Open.Refresh(OBJ_ALL_PERIODS);
m_Close.Refresh(OBJ_ALL_PERIODS);
m_High.Refresh(OBJ_ALL_PERIODS);
m_Low.Refresh(OBJ_ALL_PERIODS);
if(m_useVolumes)
{
m_Volumes.Refresh(OBJ_ALL_PERIODS);
}
if(m_useTime)
{
m_Time.Refresh(OBJ_ALL_PERIODS);
}
// Unconditional - see the matching BufferResize() comment above.
m_ATR.Refresh(OBJ_ALL_PERIODS);
m_ADZigZag.Refresh(OBJ_ALL_PERIODS);
if(m_useADCumulativeDelta)
{
m_ADCumulativeDelta.Refresh(OBJ_ALL_PERIODS);
}
if(m_useADShorteningOfThrust)
{
m_ADShorteningOfThrust.Refresh(OBJ_ALL_PERIODS);
}
if(m_useADWyckoffEventStream)
{
m_ADWyckoffEventStream.Refresh(OBJ_ALL_PERIODS);
}
if(m_useADWyckoffFailedStructure)
{
m_ADWyckoffFailedStructure.Refresh(OBJ_ALL_PERIODS);
}
if(m_useADWyckoffSignificantBarInversion)
{
m_ADWyckoffSignificantBarInversion.Refresh(OBJ_ALL_PERIODS);
}
return true;
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
//+------------------------------------------------------------------+
//| Cache-or-compute wrapper around BufferTempDataCompute(): a given |
//| now-relative bar index's feature vector is invariant until the |
//| next candle close (see m_featureCache's declaration comment), so |
//| a cache hit just replays the m_neuronsCount values already |
//| computed for this idx straight into TempData instead of re- |
//| deriving them from price/ATR/AD-indicator buffers again. |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::BufferTempData(int idx)
{
int width = m_neuronsCount;
bool cacheable = (idx >= 0 && idx < ArraySize(m_featureCacheHasValue) && width > 0);
if(cacheable && m_featureCacheHasValue[idx])
{
if(!m_featureCacheValid[idx])
return false;
int base = idx * width;
for(int f = 0; f < width; f++)
if(!TempData.Add(m_featureCache[base + f]))
return false;
return true;
}
int startTotal = TempData.Total();
bool ok = BufferTempDataCompute(idx);
if(cacheable)
{
m_featureCacheHasValue[idx] = true;
m_featureCacheValid[idx] = ok;
if(ok)
{
int base = idx * width;
int count = TempData.Total() - startTotal;
for(int f = 0; f < count && f < width; f++)
m_featureCache[base + f] = TempData.At(startTotal + f);
}
}
return ok;
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::BufferTempDataCompute(int idx)
{
double open = m_Open.GetData(idx);
double close = m_Close.GetData(idx);
double high = m_High.GetData(idx);
double low = m_Low.GetData(idx);
MqlDateTime sTime;
TimeToStruct(m_Time.GetData(idx), sTime);
if(open == EMPTY_VALUE)
return false;
// ATR-normalize every raw-price-unit feature below instead of feeding e.g. 0.0005 on EURUSD vs.
// 50.0 on a JPY pair or an index straight into the network - with Adam and hardcoded, scale-
// sensitive activations (TANH saturates, PRELU's 0.01 leak only means anything relative to the
// input's own scale), an unnormalized feature either vanishes into rounding noise or dominates
// the weighted sum depending on which symbol/timeframe happens to be loaded. Dividing by the
// bar's own ATR expresses every price-based feature as "fraction of typical volatility", which
// is comparable across symbols/timeframes and centered near zero. No ATR reading yet (e.g. the
// first few bars of history) means every price feature this bar would be meaningless - reject
// the bar via the same "return false" convention as the EMPTY_VALUE check above.
double atr = m_ATR.Main(idx);
if(atr <= 0.0 || atr == EMPTY_VALUE)
return false;
if(!TempData.Add((close - open) / atr) ||
!TempData.Add((high - open) / atr) ||
!TempData.Add((low - open) / atr) ||
// Explicit bullish/bearish flag - (close-open)/atr already encodes direction *and* magnitude
// together, which asks the network to disentangle "which way" from "how much" out of a single
// continuous value. Giving direction its own clean +1/-1/0 signal removes that ambiguity.
!TempData.Add(close > open ? 1.0 : (close < open ? -1.0 : 0.0)))
{
return false;
}
if(m_useSwingContext)
{
// Most recent CONFIRMED swing pivot as of bar idx - "confirmed" meaning at least
// m_swingConfirmationBars MORE bars have closed after it, the exact same repainting embargo
// AdvanceZigZagLabelState() applies on the label side (see m_swingConfirmationBars' and
// m_useSwingContext's declaration comments). Skipping this embargo here - e.g. reading
// m_ADZigZag's raw current buffer value instead - would leak information a live bar at idx
// could never actually have had yet, since ZigZag's most recent 1-3 legs are still provisional
// and can be revised as new bars arrive.
int pivotIdx = -1;
double pivotPrice = 0.0;
bool pivotIsLow = false;
if(!FindConfirmedZigZagPivot(idx + MathMax(m_swingConfirmationBars, 1), pivotIdx, pivotPrice, pivotIsLow))
{
// No confirmed pivot within the scan cap (e.g. right at the start of available history) -
// this is legitimately "no swing context yet", not bad/missing data, so a neutral 0-fill
// keeps the bar usable rather than rejecting it outright like the ATR/EMPTY_VALUE guards do.
if(!TempData.Add(0.0) || !TempData.Add(0.0) || !TempData.Add(0.0) || !TempData.Add(0.0) || !TempData.Add(0.0))
return false;
}
else
{
// Direction of the CURRENT leg: the last confirmed pivot being a bottom means price has been
// rising away from it (an up-leg) ever since, and vice versa - same +1/-1 convention as the
// bullish/bearish flag above, just at swing scale instead of single-bar scale.
double direction = pivotIsLow ? 1.0 : -1.0;
// How far price has travelled since that pivot, ATR-normalized and signed (+ve above the
// pivot price, -ve below) - clamped generously since an extended trending leg has no natural
// ceiling the way a single bar's range does.
double distSincePivot = MathMax(-10.0, MathMin(10.0, (close - pivotPrice) / atr));
// Magnitude of the PRIOR completed leg (the pivot immediately before pivotIdx) - a scale
// reference for "is the current move big or small relative to the last full swing". No
// additional embargo needed here (see FindConfirmedZigZagPivot()'s declaration comment) -
// anything at or before an already-confirmed pivot is necessarily even older.
int priorPivotIdx = -1;
double priorPivotPrice = 0.0;
bool priorPivotIsLow = false;
bool havePrior = FindConfirmedZigZagPivot(pivotIdx + 1, priorPivotIdx, priorPivotPrice, priorPivotIsLow);
double priorLegMagnitude = havePrior ? MathMax(0.0, MathMin(10.0, MathAbs(pivotPrice - priorPivotPrice) / atr)) : 0.0;
// Retracement/extension ratio (current distance relative to the prior leg's own size) -
// Fibonacci-style relative position, often more informative than either raw magnitude alone
// since it's comparable across both quiet and volatile regimes. 0 when there's no prior leg
// to compare against yet.
double retracementRatio = (havePrior && priorLegMagnitude > 0.0001) ?
MathMax(-5.0, MathMin(5.0, distSincePivot / priorLegMagnitude)) : 0.0;
// Swing age (bars since the pivot) - a maturity/exhaustion proxy, same +/- style clamp
// convention as the volume-ratio feature below.
double barsSincePivot = MathMax(0.0, MathMin(5.0, (double)(pivotIdx - idx) / 100.0));
if(!TempData.Add(direction) ||
!TempData.Add(distSincePivot) ||
!TempData.Add(priorLegMagnitude) ||
!TempData.Add(retracementRatio) ||
!TempData.Add(barsSincePivot))
return false;
}
}
if(m_useVolumes)
{
// Relative change, not raw tick/volume delta - trading activity magnitude also varies wildly
// across symbols/timeframes, so the previous bar's own volume is used as the scale reference,
// same logic as ATR-normalizing price above. Guard against a zero previous-bar volume (e.g. a
// holiday-thin session) instead of dividing by it. Clamped to +/-5 - unlike the ATR-normalized
// price features, this ratio has no natural ceiling (a thin previous bar, e.g. 1 tick, followed
// by a normal one produces a huge outlier), and every other feature fed into the network is
// already bounded to roughly a single-digit range.
double prevVolume = m_Volumes.Main(idx + 1);
double volumeDelta = m_Volumes.Main(idx) - prevVolume;
double volumeChangeRatio = prevVolume > 0.0 ? volumeDelta / prevVolume : 0.0;
if(!TempData.Add(MathMax(-5.0, MathMin(5.0, volumeChangeRatio))))
return false;
}
if(m_useTime)
{
// Normalize time (cyclical encoding)
if(!TempData.Add(sin(2 * M_PI * sTime.hour / 24.0)))
return false;
if(!TempData.Add(cos(2 * M_PI * sTime.hour / 24.0)))
return false;
if(!TempData.Add(sin(2 * M_PI * sTime.day_of_week / 7.0)))
return false;
if(!TempData.Add(cos(2 * M_PI * sTime.day_of_week / 7.0)))
return false;
if(!TempData.Add(sin(2 * M_PI * sTime.mon / 12.0)))
return false;
if(!TempData.Add(cos(2 * M_PI * sTime.mon / 12.0)))
return false;
}
if(m_useATR)
{
// ATR/close (volatility as a fraction of price), not raw ATR - the raw absolute value is
// itself unnormalized (e.g. ~0.0012 on EURUSD vs. ~1.5 on gold, and drifts over time even on
// one symbol as its price level changes), which is exactly the kind of scale-dependent
// feature this whole normalization pass is fixing everywhere else.
if(!TempData.Add(close != 0.0 ? atr / close : 0.0))
return false;
}
if(m_useNews)
{
// Event proximity + impact only - see this member's declaration comment and
// System\NewsRelevance.mqh's ImpactWeightedProximity() for why the forward-looking half
// (searchForward=true) isn't lookahead bias despite being computed for a historical bar.
datetime barTime = m_Time.GetData(idx);
double newsRecency = ImpactWeightedProximity(m_symbol.Name(), barTime, m_newsFeatureWindowMinutes, false);
double newsProximity = ImpactWeightedProximity(m_symbol.Name(), barTime, m_newsFeatureWindowMinutes, true);
if(!TempData.Add(newsRecency) || !TempData.Add(newsProximity))
return false;
}
if(m_useADCumulativeDelta)
{
// buffers: 0=Pressure, 1=CumulativeDelta, 2=BullishPressure, 3=BearishPressure, 4=Absorption, 5=Initiative.
// CumulativeDelta (buffer 1) is now cumulativeDelta/sumVolume clamped +/-2 (same scale as every
// other buffer here, see ADCumulativeDelta.mq5) - a pure order-flow-imbalance ratio, distinct from
// Pressure (buffer 0), which is this same term further adjusted by Initiative/Absorption.
if(!TempData.Add(m_ADCumulativeDelta.GetData(0, idx)) || // Pressure
!TempData.Add(m_ADCumulativeDelta.GetData(1, idx)) || // CumulativeDelta
!TempData.Add(m_ADCumulativeDelta.GetData(2, idx)) || // BullishPressure
!TempData.Add(m_ADCumulativeDelta.GetData(3, idx)) || // BearishPressure
!TempData.Add(m_ADCumulativeDelta.GetData(4, idx)) || // Absorption
!TempData.Add(m_ADCumulativeDelta.GetData(5, idx))) // Initiative
return false;
}
if(m_useADShorteningOfThrust)
{
// buffers: 0=SOT, 1=SOTEffortRegime, 2=SOTConfirmation, 3=SOTPushRegime
if(!TempData.Add(m_ADShorteningOfThrust.GetData(0, idx)) || // SOT
!TempData.Add(m_ADShorteningOfThrust.GetData(1, idx)) || // SOTEffortRegime
!TempData.Add(m_ADShorteningOfThrust.GetData(2, idx)) || // SOTConfirmation
!TempData.Add(m_ADShorteningOfThrust.GetData(3, idx))) // SOTPushRegime
return false;
}
if(m_useADWyckoffEventStream)
{
// buffers: 0=EventCode, 1=EventPhase, 2=ZoneTop, 3=ZoneBottom, 4=EventPrice, 5=StructuralPhase,
// 6=CHoCHTrendToRange, 7=CHoCHRangeToTrend, 8=SlopeAccumulationBullish, 9=SlopeAccumulationBearish,
// 10=SlopeDistributionBullish, 11=SlopeDistributionBearish, 12=Reaccumulation, 13=Redistribution.
// Buffer 4 (EventPrice) is deliberately skipped below - per the indicator's own source
// (ADWyckoffEventStream.mq5: "BufColor[wi]=(ev!=0)?C[i]:0;"), it's just this bar's close price
// echoed back when an event fires (0 otherwise), kept only so a charting/backtesting tool like
// StrategyQuant can anchor an arrow to a price. It carries no information the network doesn't
// already have (EventCode already flags whether an event fired; the close is already in the
// base OHLC features), and normalizing it as a "distance from close" like ZoneTop/ZoneBottom
// below would be actively wrong: it's close-close=0 on event bars but 0-close=-close (a raw,
// ATR-blown-up price) on every other bar - a huge, meaningless outlier feature.
// Buffer 1 (EventPhase) is also deliberately skipped - per the indicator's own source
// (ADWyckoffEventStream.mq5: "BufPhase[wi]=(double)ev;", the exact same `ev` variable written to
// BufEvent one line above), it's a byte-for-byte duplicate of EventCode, not a distinct phase
// reading - the real phase signal is StructuralPhase (buffer 5), already included below.
if(!TempData.Add(m_ADWyckoffEventStream.GetData(0, idx)) || // EventCode
!TempData.Add((m_ADWyckoffEventStream.GetData(2, idx) - close) / atr) || // ZoneTop
!TempData.Add((m_ADWyckoffEventStream.GetData(3, idx) - close) / atr) || // ZoneBottom
!TempData.Add(m_ADWyckoffEventStream.GetData(5, idx)) || // StructuralPhase
!TempData.Add(m_ADWyckoffEventStream.GetData(6, idx)) || // CHoCHTrendToRange
!TempData.Add(m_ADWyckoffEventStream.GetData(7, idx)) || // CHoCHRangeToTrend
!TempData.Add(m_ADWyckoffEventStream.GetData(8, idx)) || // SlopeAccumulationBullish
!TempData.Add(m_ADWyckoffEventStream.GetData(9, idx)) || // SlopeAccumulationBearish
!TempData.Add(m_ADWyckoffEventStream.GetData(10, idx)) || // SlopeDistributionBullish
!TempData.Add(m_ADWyckoffEventStream.GetData(11, idx)) || // SlopeDistributionBearish
!TempData.Add(m_ADWyckoffEventStream.GetData(12, idx)) || // Reaccumulation
!TempData.Add(m_ADWyckoffEventStream.GetData(13, idx))) // Redistribution
return false;
}
if(m_useADWyckoffFailedStructure)
{
// buffers: 0=Value, 1=BullishStructuralFailure, 2=BearishStructuralFailure, 3=FailedAccumulation, 4=FailedDistribution
if(!TempData.Add(m_ADWyckoffFailedStructure.GetData(0, idx)) || // Value
!TempData.Add(m_ADWyckoffFailedStructure.GetData(1, idx)) || // BullishStructuralFailure
!TempData.Add(m_ADWyckoffFailedStructure.GetData(2, idx)) || // BearishStructuralFailure
!TempData.Add(m_ADWyckoffFailedStructure.GetData(3, idx)) || // FailedAccumulation
!TempData.Add(m_ADWyckoffFailedStructure.GetData(4, idx))) // FailedDistribution
return false;
}
if(m_useADWyckoffSignificantBarInversion)
{
// buffers: 0=SignificantBarQuality, 1=BullishSignificantBar, 2=BearishSignificantBar, 3=BullishControlFlip, 4=BearishControlFlip
if(!TempData.Add(m_ADWyckoffSignificantBarInversion.GetData(0, idx)) || // SignificantBarQuality
!TempData.Add(m_ADWyckoffSignificantBarInversion.GetData(1, idx)) || // BullishSignificantBar
!TempData.Add(m_ADWyckoffSignificantBarInversion.GetData(2, idx)) || // BearishSignificantBar
!TempData.Add(m_ADWyckoffSignificantBarInversion.GetData(3, idx)) || // BullishControlFlip
!TempData.Add(m_ADWyckoffSignificantBarInversion.GetData(4, idx))) // BearishControlFlip
return false;
}
return true;
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::SaveTopologyConfiguration(string fileName, int initialNeuronsCount, int hiddenLayersCount, double neuronsReduction, int minNeuronsCount, int optimizationAlgo, int historyBars, int outputNeuronsCount, int neuronsCount, int studyPeriod, int minTrainYear, bool isInitialized, int stopTrainWR, int fractalPeriods, bool common)
{
string configFileName = fileName + ".cfg";
int handle = FileOpen(configFileName, FILE_WRITE | FILE_BIN | (common ? FILE_COMMON : 0));
if(handle == INVALID_HANDLE)
{
Print("Error: Unable to open file ", configFileName);
ResetLastError();
return false;
}
if(FileWriteInteger(handle, initialNeuronsCount) < sizeof(int))
{
Print("Error writing initialNeuronsCount : Error code: ", GetLastError());
ResetLastError();
return false;
}
if(FileWriteInteger(handle, hiddenLayersCount) < sizeof(int))
{
Print("Error writing hiddenLayersCount : Error code: ", GetLastError());
ResetLastError();
return false;
}
if(FileWriteDouble(handle, neuronsReduction) < sizeof(double))
{
Print("Error writing neuronsReduction : Error code: ", GetLastError());
ResetLastError();
return false;
}
if(FileWriteInteger(handle, minNeuronsCount) < sizeof(int))
{
Print("Error writing minNeuronsCount : Error code: ", GetLastError());
ResetLastError();
return false;
}
if(FileWriteInteger(handle, optimizationAlgo) < sizeof(int))
{
Print("Error writing optimizationAlgo : Error code: ", GetLastError());
ResetLastError();
return false;
}
if(FileWriteInteger(handle, historyBars) < sizeof(int))
{
Print("Error writing historyBars : Error code: ", GetLastError());
ResetLastError();
return false;
}
if(FileWriteInteger(handle, outputNeuronsCount) < sizeof(int))
{
Print("Error writing outputNeuronsCount : Error code: ", GetLastError());
ResetLastError();
return false;
}
if(FileWriteInteger(handle, neuronsCount) < sizeof(int))
{
Print("Error writing neuronsCount : Error code: ", GetLastError());
ResetLastError();
return false;
}
if(FileWriteInteger(handle, studyPeriod) < sizeof(int))
{
Print("Error writing studyPeriod : Error code: ", GetLastError());
ResetLastError();
return false;
}
if(FileWriteInteger(handle, minTrainYear) < sizeof(int))
{
Print("Error writing minTrainYear : Error code: ", GetLastError());
ResetLastError();
return false;
}
if(FileWriteInteger(handle, isInitialized) < sizeof(int))
{
Print("Error writing isInitialized : Error code: ", GetLastError());
ResetLastError();
return false;
}
if(FileWriteInteger(handle, stopTrainWR) < sizeof(int))
{
Print("Error writing stopTrainWR : Error code: ", GetLastError());
ResetLastError();
return false;
}
if(FileWriteInteger(handle, fractalPeriods) < sizeof(int))
{
Print("Error writing fractalPeriods : Error code: ", GetLastError());
ResetLastError();
return false;
}
FileClose(handle);
return true;
}
//+------------------------------------------------------------------+
//| |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::LoadAndCompareTopologyConfiguration(string fileName, int initialNeuronsCount, int hiddenLayersCount, double neuronsReduction, int minNeuronsCount, int optimizationAlgo, int historyBars, int outputNeuronsCount, int neuronsCount, int studyPeriod, int minTrainYear, bool isInitialized, int stopTrainWR, int fractalPeriods, bool common)
{
string configFileName = fileName + ".cfg";
if(!FileIsExist(configFileName, common ? FILE_COMMON : 0))
{
Print("Error: File ", configFileName, " does not exist");
return false;
}
int handle = FileOpen(configFileName, FILE_READ | FILE_BIN | (common ? FILE_COMMON : 0));
if(handle == INVALID_HANDLE)
{
Print("Error: Unable to open file ", configFileName);
return false;
}
int savedInitialNeurons = FileReadInteger(handle);
int savedHiddenLayers = FileReadInteger(handle);
double savedReductionFactor = FileReadDouble(handle);
int savedMinNeurons = FileReadInteger(handle);
int savedOptimizationAlgo = FileReadInteger(handle);
int savedHistoryBars = FileReadInteger(handle);
int savedOutputNeuronsCount = FileReadInteger(handle);
int savedNeuronsCount = FileReadInteger(handle);
int savedStudyPeriod = FileReadInteger(handle);
int savedMinTrainYear = FileReadInteger(handle);
bool savedIsInitialized = FileReadInteger(handle);
int savedStopTrainWR = FileReadInteger(handle);
int savedFractalPeriods = FileReadInteger(handle);
FileClose(handle);
if(savedInitialNeurons != initialNeuronsCount ||
savedHiddenLayers != hiddenLayersCount ||
savedReductionFactor != neuronsReduction ||
savedMinNeurons != minNeuronsCount ||
savedOptimizationAlgo != optimizationAlgo ||
savedHistoryBars != historyBars ||
savedOutputNeuronsCount != outputNeuronsCount ||
savedNeuronsCount != neuronsCount ||
savedStudyPeriod != studyPeriod ||
savedMinTrainYear != minTrainYear ||
savedIsInitialized != isInitialized ||
savedStopTrainWR != stopTrainWR ||
savedFractalPeriods != fractalPeriods)
{
Print("Error: Configuration mismatch. Deleting file ", configFileName);
FileDelete(configFileName, common ? FILE_COMMON : 0);
return false;
}
return true;
}
//+------------------------------------------------------------------+
//| PurgeChart - Removes all visual objects from the current chart |
//+------------------------------------------------------------------+
void CExpertSignalAIBase::PurgeChart(void)
{
// Specify the current chart ID (0 for the current chart)
long chartID = 0;
// Remove all objects of any type (use specific type if needed)
ObjectsDeleteAll(chartID);
}
//+------------------------------------------------------------------+
//| Initialize Open indicators. |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::InitOpen(CIndicators * indicators)
{
//--- check pointer
if(indicators == NULL)
return (false);
//--- add object to collection
if(!indicators.Add(GetPointer(m_Open)))
{
printf(__FUNCTION__ + ": error adding object");
return (false);
}
//--- initialize object
if(!m_Open.Create(m_symbol.Name(), m_period))
{
printf(__FUNCTION__ + ": error initializing object");
return (false);
}
//--- ok
return (true);
}
//+------------------------------------------------------------------+
//| Initialize Close indicators. |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::InitClose(CIndicators * indicators)
{
//--- check pointer
if(indicators == NULL)
return (false);
//--- add object to collection
if(!indicators.Add(GetPointer(m_Close)))
{
printf(__FUNCTION__ + ": error adding object");
return (false);
}
//--- initialize object
if(!m_Close.Create(m_symbol.Name(), m_period))
{
printf(__FUNCTION__ + ": error initializing object");
return (false);
}
//--- ok
return (true);
}
//+------------------------------------------------------------------+
//| Initialize High indicators. |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::InitHigh(CIndicators * indicators)
{
//--- check pointer
if(indicators == NULL)
return (false);
//--- add object to collection
if(!indicators.Add(GetPointer(m_High)))
{
printf(__FUNCTION__ + ": error adding object");
return (false);
}
//--- initialize object
if(!m_High.Create(m_symbol.Name(), m_period))
{
printf(__FUNCTION__ + ": error initializing object");
return (false);
}
//--- ok
return (true);
}
//+------------------------------------------------------------------+
//| Initialize Low indicators. |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::InitLow(CIndicators * indicators)
{
//--- check pointer
if(indicators == NULL)
return (false);
//--- add object to collection
if(!indicators.Add(GetPointer(m_Low)))
{
printf(__FUNCTION__ + ": error adding object");
return (false);
}
//--- initialize object
if(!m_Low.Create(m_symbol.Name(), m_period))
{
printf(__FUNCTION__ + ": error initializing object");
return (false);
}
//--- ok
return (true);
}
//+------------------------------------------------------------------+
//| Initialize Time indicators. |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::InitTime(CIndicators * indicators)
{
//--- check pointer
if(indicators == NULL)
return (false);
//--- add object to collection
if(!indicators.Add(GetPointer(m_Time)))
{
printf(__FUNCTION__ + ": error adding object");
return (false);
}
//--- initialize object
if(!m_Time.Create(m_symbol.Name(), m_period))
{
printf(__FUNCTION__ + ": error initializing object");
return (false);
}
//--- ok
return (true);
}
//+------------------------------------------------------------------+
//| Initialize Volumes indicators. |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::InitVolumes(CIndicators * indicators)
{
//--- check pointer
if(indicators == NULL)
return (false);
//--- add object to collection
if(!indicators.Add(GetPointer(m_Volumes)))
{
printf(__FUNCTION__ + ": error adding object");
return (false);
}
//--- initialize object
if(!m_Volumes.Create(m_symbol.Name(), m_period, VolumeData))
{
printf(__FUNCTION__ + ": error initializing object");
return (false);
}
//--- ok
return (true);
}
//+------------------------------------------------------------------+
//| Initialize AD Cumulative Delta (CustomIndicators\ADCumulativeDelta.mq5) |
//| Loaded via iCustom/CiCustom, not a built-in Ci* class - the compiled |
//| indicator must be present under MQL5\Indicators\ (see |
//| ExtractCustomIndicators() in Warrior_EA.mq5). Uses the indicator's |
//| own input defaults; 6 output buffers, one TempData feature each. |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::InitADCumulativeDelta(CIndicators * indicators, bool addToCollection)
{
//--- check pointer
if(indicators == NULL)
return (false);
//--- add object to collection
if(addToCollection && !indicators.Add(GetPointer(m_ADCumulativeDelta)))
{
printf(__FUNCTION__ + ": error adding object");
return (false);
}
//--- initialize object; params[1..] mirror ADCumulativeDelta.mq5's own input order exactly
MqlParam params[11];
params[0].type = TYPE_STRING;
params[0].string_value = "ADCumulativeDelta";
params[1].type = TYPE_INT;
params[1].integer_value = m_indicatorTuner.adCumDelta.lookback; // InpLookbackPeriod
params[2].type = TYPE_DOUBLE;
params[2].double_value = m_indicatorTuner.adCumDelta.volClimax; // InpVolumeClimaxMultiplier
params[3].type = TYPE_DOUBLE;
params[3].double_value = m_indicatorTuner.adCumDelta.volHigh; // InpVolumeHighMultiplier
params[4].type = TYPE_DOUBLE;
params[4].double_value = m_indicatorTuner.adCumDelta.rangeClimax; // InpRangeClimaxMultiplier
params[5].type = TYPE_DOUBLE;
params[5].double_value = m_indicatorTuner.adCumDelta.rangeSignificant; // InpRangeSignificantMult
params[6].type = TYPE_DOUBLE;
params[6].double_value = m_indicatorTuner.adCumDelta.stVolRatio; // InpSTVolumeRatio
params[7].type = TYPE_DOUBLE;
params[7].double_value = m_indicatorTuner.adCumDelta.atrMult; // InpATRMultiplier
params[8].type = TYPE_INT;
params[8].integer_value = 0; // InpContextMode - DO NOT tune
params[9].type = TYPE_INT;
params[9].integer_value = 5; // InpSessionType - DO NOT tune
params[10].type = TYPE_INT;
params[10].integer_value = 1; // InpSessionCount - DO NOT tune
if(!m_ADCumulativeDelta.Create(m_symbol.Name(), m_period, IND_CUSTOM, 11, params))
{
printf(__FUNCTION__ + ": error initializing object");
return (false);
}
m_ADCumulativeDelta.NumBuffers(6);
//--- ok
return (true);
}
//+------------------------------------------------------------------+
//| Initialize AD Shortening of Thrust (CustomIndicators\ADShorteningOfThrust.mq5) |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::InitADShorteningOfThrust(CIndicators * indicators, bool addToCollection)
{
//--- check pointer
if(indicators == NULL)
return (false);
//--- add object to collection
if(addToCollection && !indicators.Add(GetPointer(m_ADShorteningOfThrust)))
{
printf(__FUNCTION__ + ": error adding object");
return (false);
}
//--- initialize object; params[1..] mirror ADShorteningOfThrust.mq5's own input order exactly
MqlParam params[7];
params[0].type = TYPE_STRING;
params[0].string_value = "ADShorteningOfThrust";
params[1].type = TYPE_INT;
params[1].integer_value = m_indicatorTuner.adSOT.thrustLookback; // InpThrustLookback
params[2].type = TYPE_INT;
params[2].integer_value = m_indicatorTuner.adSOT.minImpulses; // InpMinImpulses
params[3].type = TYPE_DOUBLE;
params[3].double_value = m_indicatorTuner.adSOT.sotThreshold; // InpSOTThreshold
params[4].type = TYPE_INT;
params[4].integer_value = 0; // InpContextMode - DO NOT tune
params[5].type = TYPE_INT;
params[5].integer_value = 5; // InpSessionType - DO NOT tune
params[6].type = TYPE_INT;
params[6].integer_value = 1; // InpSessionCount - DO NOT tune
if(!m_ADShorteningOfThrust.Create(m_symbol.Name(), m_period, IND_CUSTOM, 7, params))
{
printf(__FUNCTION__ + ": error initializing object");
return (false);
}
m_ADShorteningOfThrust.NumBuffers(4);
//--- ok
return (true);
}
//+------------------------------------------------------------------+
//| Initialize AD Wyckoff Event Stream (CustomIndicators\ADWyckoffEventStream.mq5) |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::InitADWyckoffEventStream(CIndicators * indicators, bool addToCollection)
{
//--- check pointer
if(indicators == NULL)
return (false);
//--- add object to collection
if(addToCollection && !indicators.Add(GetPointer(m_ADWyckoffEventStream)))
{
printf(__FUNCTION__ + ": error adding object");
return (false);
}
//--- initialize object; params[1..] mirror ADWyckoffEventStream.mq5's own input order exactly
MqlParam params[12];
params[0].type = TYPE_STRING;
params[0].string_value = "ADWyckoffEventStream";
params[1].type = TYPE_INT;
params[1].integer_value = m_indicatorTuner.adWES.lookback; // InpLookback
params[2].type = TYPE_INT;
params[2].integer_value = m_indicatorTuner.adWES.zigzag; // InpZigZag
params[3].type = TYPE_DOUBLE;
params[3].double_value = m_indicatorTuner.adWES.volClimax; // InpVolClimax
params[4].type = TYPE_DOUBLE;
params[4].double_value = m_indicatorTuner.adWES.volHigh; // InpVolHigh
params[5].type = TYPE_DOUBLE;
params[5].double_value = m_indicatorTuner.adWES.rangeClimax; // InpRangeClimax
params[6].type = TYPE_DOUBLE;
params[6].double_value = m_indicatorTuner.adWES.rangeSignificant; // InpRangeSignificant
params[7].type = TYPE_DOUBLE;
params[7].double_value = m_indicatorTuner.adWES.stVolRatio; // InpSTVolRatio
params[8].type = TYPE_DOUBLE;
params[8].double_value = m_indicatorTuner.adWES.atr; // InpATR
params[9].type = TYPE_INT;
params[9].integer_value = 0; // InpContextMode - DO NOT tune
params[10].type = TYPE_INT;
params[10].integer_value = 5; // InpSessionType - DO NOT tune
params[11].type = TYPE_INT;
params[11].integer_value = 1; // InpSessionCount - DO NOT tune
if(!m_ADWyckoffEventStream.Create(m_symbol.Name(), m_period, IND_CUSTOM, 12, params))
{
printf(__FUNCTION__ + ": error initializing object");
return (false);
}
m_ADWyckoffEventStream.NumBuffers(14);
//--- ok
return (true);
}
//+------------------------------------------------------------------+
//| Initialize AD Wyckoff Failed Structure (CustomIndicators\ADWyckoffFailedStructure.mq5) |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::InitADWyckoffFailedStructure(CIndicators * indicators, bool addToCollection)
{
//--- check pointer
if(indicators == NULL)
return (false);
//--- add object to collection
if(addToCollection && !indicators.Add(GetPointer(m_ADWyckoffFailedStructure)))
{
printf(__FUNCTION__ + ": error adding object");
return (false);
}
//--- initialize object; params[1..] mirror ADWyckoffFailedStructure.mq5's own input order exactly
MqlParam params[12];
params[0].type = TYPE_STRING;
params[0].string_value = "ADWyckoffFailedStructure";
params[1].type = TYPE_INT;
params[1].integer_value = m_indicatorTuner.adWFS.lookback; // InpLookbackPeriod
params[2].type = TYPE_INT;
params[2].integer_value = m_indicatorTuner.adWFS.zigzagStrength; // InpZigZagStrength
params[3].type = TYPE_DOUBLE;
params[3].double_value = m_indicatorTuner.adWFS.volClimax; // InpVolumeClimaxMultiplier
params[4].type = TYPE_DOUBLE;
params[4].double_value = m_indicatorTuner.adWFS.volHigh; // InpVolumeHighMultiplier
params[5].type = TYPE_DOUBLE;
params[5].double_value = m_indicatorTuner.adWFS.rangeClimax; // InpRangeClimaxMultiplier
params[6].type = TYPE_DOUBLE;
params[6].double_value = m_indicatorTuner.adWFS.rangeSignificant; // InpRangeSignificantMult
params[7].type = TYPE_DOUBLE;
params[7].double_value = m_indicatorTuner.adWFS.stVolRatio; // InpSTVolumeRatio
params[8].type = TYPE_DOUBLE;
params[8].double_value = m_indicatorTuner.adWFS.atrMult; // InpATRMultiplier
params[9].type = TYPE_INT;
params[9].integer_value = 0; // InpContextMode - DO NOT tune
params[10].type = TYPE_INT;
params[10].integer_value = 5; // InpSessionType - DO NOT tune
params[11].type = TYPE_INT;
params[11].integer_value = 1; // InpSessionCount - DO NOT tune
if(!m_ADWyckoffFailedStructure.Create(m_symbol.Name(), m_period, IND_CUSTOM, 12, params))
{
printf(__FUNCTION__ + ": error initializing object");
return (false);
}
m_ADWyckoffFailedStructure.NumBuffers(5);
//--- ok
return (true);
}
//+------------------------------------------------------------------+
//| Initialize AD Wyckoff Significant Bar Inversion (CustomIndicators\ADWyckoffSignificantBarInversion.mq5) |
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::InitADWyckoffSignificantBarInversion(CIndicators * indicators, bool addToCollection)
{
//--- check pointer
if(indicators == NULL)
return (false);
//--- add object to collection
if(addToCollection && !indicators.Add(GetPointer(m_ADWyckoffSignificantBarInversion)))
{
printf(__FUNCTION__ + ": error adding object");
return (false);
}
//--- initialize object; params[1..] mirror ADWyckoffSignificantBarInversion.mq5's own input order exactly
MqlParam params[8];
params[0].type = TYPE_STRING;
params[0].string_value = "ADWyckoffSignificantBarInversion";
params[1].type = TYPE_INT;
params[1].integer_value = m_indicatorTuner.adWSBI.lookback; // InpLookback
params[2].type = TYPE_DOUBLE;
params[2].double_value = m_indicatorTuner.adWSBI.rangeSignificant; // InpRangeSignificant
params[3].type = TYPE_DOUBLE;
params[3].double_value = m_indicatorTuner.adWSBI.volumeHigh; // InpVolumeHigh
params[4].type = TYPE_DOUBLE;
params[4].double_value = m_indicatorTuner.adWSBI.atr; // InpATR
params[5].type = TYPE_INT;
params[5].integer_value = 0; // InpContextMode - DO NOT tune
params[6].type = TYPE_INT;
params[6].integer_value = 5; // InpSessionType - DO NOT tune
params[7].type = TYPE_INT;
params[7].integer_value = 1; // InpSessionCount - DO NOT tune
if(!m_ADWyckoffSignificantBarInversion.Create(m_symbol.Name(), m_period, IND_CUSTOM, 8, params))
{
printf(__FUNCTION__ + ": error initializing object");
return (false);
}
m_ADWyckoffSignificantBarInversion.NumBuffers(5);
//--- ok
return (true);
}
//+------------------------------------------------------------------+
//| Initialize AD ZigZag (CustomIndicators\ADZigZag.mq5) - the |
//| training-label source (see m_ADZigZag's declaration comment). |
//| Always run at its own stock defaults (Depth=12, Deviation=5, |
//| Backstep=3) - unlike the AD* feature indicators above, this has |
//| no tunable-param struct and is never touched by AutoTuneIndicators.|
//+------------------------------------------------------------------+
bool CExpertSignalAIBase::InitADZigZag(CIndicators * indicators, bool addToCollection)
{
//--- check pointer
if(indicators == NULL)
return (false);
//--- add object to collection
if(addToCollection && !indicators.Add(GetPointer(m_ADZigZag)))
{
printf(__FUNCTION__ + ": error adding object");
return (false);
}
//--- initialize object; params[1..] mirror ADZigZag.mq5's own input order exactly - stock defaults,
//--- intentionally not sourced from a tunable params struct (see this function's declaration comment)
MqlParam params[4];
params[0].type = TYPE_STRING;
params[0].string_value = "ADZigZag";
params[1].type = TYPE_INT;
params[1].integer_value = 12; // InpDepth
params[2].type = TYPE_INT;
params[2].integer_value = 5; // InpDeviation
params[3].type = TYPE_INT;
params[3].integer_value = 3; // InpBackstep
if(!m_ADZigZag.Create(m_symbol.Name(), m_period, IND_CUSTOM, 4, params))
{
printf(__FUNCTION__ + ": error initializing object");
return (false);
}
// Must match ADZigZag.mq5's #property indicator_buffers exactly (3: main ZigZag buffer + 2
// internal INDICATOR_CALCULATIONS buffers), even though only buffer 0 is ever read via
// GetData() - see the working AD Wyckoff indicators' InitAD*() for the same pattern.
m_ADZigZag.NumBuffers(3);
//--- ok
return (true);
}
//+------------------------------------------------------------------+
View git blame Copy permalink