forked from animatedread/Warrior_EA
444909d0a3
The NN now has a target that is not per-bar direction (closed, best-of-999 p=1.0000): P(win | this journaled candidate, at the EA's own SL/TP, net of cost). One net for all 52 pattern-sides, AIType=AI_META. - NetForward.mqh: the host-side softmax+CE gradient generalized total==3 -> 2||3 on both backprop paths; a 2-class softmax IS a logistic head, and no compute backend changes. - SignalMETA.mqh (new): corpus loaded read-only from the LARGEST signal DB on disk (decoupled from the config fingerprint that burned four S1 runs); the GMT->server offset is measured PER ROW against entryPrice vs bar open (DST-immune, histogram logged); a window-span regime filter drops the pre-2017 daily-backfill rows; 31-feature setup descriptor appended at the input (26 one-hot + side + tanh netVote + SL/TP ATR + spread/ATR). - Training.mqh: candidate-queued pass 1, binary-target pass 2, per-candidate calibration (2.5) and OOS (3) walks. Counter mapping win->Buy / loss->Sell lets checkpoint selection, the edge floor, the plateau ladder and the family-wise deploy gate run UNCHANGED: precision reads as win rate among traded candidates, chance as the base win rate, recalls as sensitivity/ specificity. Era-end META line: coverage x (p - break-even) vs the null. - Labels are the side-conditional triple-barrier win caches - never the DB's stop-and-reverse outcome. Logit adjustment deliberately skipped (~40% base rate). Live inference + online learning guarded off until S3. - Fingerprint: conditional |TGT:META1; State\META\ folder + 2-output filename slot keep meta models fully separate from direction models. Compiles clean (0 errors, 0 warnings). S2 run = attach a chart with AIType=AI_META; S3 wires the votes via the per-side hooks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
630 lines
38 KiB
MQL5
630 lines
38 KiB
MQL5
//+------------------------------------------------------------------+
|
|
//| Warrior_EA |
|
|
//| AnimateDread |
|
|
//| |
|
|
//| Read-time signal production: softmax, prior calibration, class p|
|
|
//| |
|
|
//| PARTIAL IMPLEMENTATION FILE - not standalone. |
|
|
//| This holds CExpertSignalAIBase method BODIES only. The class |
|
|
//| declaration lives in Expert\ExpertSignalAIBase.mqh, which |
|
|
//| #includes this file at the bottom, after the declaration. Do not |
|
|
//| include it anywhere else and do not compile it on its own. |
|
|
//| |
|
|
//| Split out purely to make the 8216-line original navigable; the |
|
|
//| code inside was moved verbatim, not rewritten. |
|
|
//+------------------------------------------------------------------+
|
|
#ifndef WARRIOR_AIBASE_INFERENCE_MQH
|
|
#define WARRIOR_AIBASE_INFERENCE_MQH
|
|
//+------------------------------------------------------------------+
|
|
//| Post-convergence "new bar" handler - see ScheduleTrainingIfNeeded()|
|
|
//| for why this exists: once m_trainingComplete is true, a plain new |
|
|
//| bar must NOT re-enter Train()'s full era loop (which resets the |
|
|
//| best-checkpoint/eta-decay tracking and runs real Net.backProp() |
|
|
//| passes again, silently perturbing an already-converged model |
|
|
//| forever, once per bar, with no way to ever actually finish). This |
|
|
//| only refreshes the price/indicator buffers and re-runs inference |
|
|
//| for the newest bar so dPrevSignal/the chart arrow stay current - |
|
|
//| identical cost to what Train() does per-bar, minus every bit of |
|
|
//| training (label caching, backProp, checkpointing). |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::RefreshConvergedSignal(void)
|
|
{
|
|
//--- Meta target: live inference is S3's work (the meta head consumes FIRED CANDIDATES via the
|
|
//--- per-side hooks, not a bare bar window - a candidate-less forward would also be width-
|
|
//--- mismatched against the meta input layer). Until S3 lands, a trained meta model just holds.
|
|
if(IsMetaTarget())
|
|
return;
|
|
//--- Size the buffers from what the FEATURE BUILDER actually needs, not from a date delta.
|
|
//--- This used to be `Bars(sym, period, dtStudied, TimeCurrent()) + m_historyBars`. dtStudied is a
|
|
//--- training watermark, and in the Strategy Tester it is loaded from a LIVE-chart save whose
|
|
//--- timestamp is AHEAD of the simulated date - so the interval inverts, Bars() returns ~0, and the
|
|
//--- buffer came out at exactly m_historyBars. That is just deep enough for the OHLC window to
|
|
//--- succeed and far too shallow for the swing-context block behind it: the Donchian-50, the 20-bar
|
|
//--- return and the SMA extension all reach further back than m_historyBars, hit the end of the
|
|
//--- loaded series, and take their graceful degraded path. The result was silent - no error, no short
|
|
//--- window, just inference computing DIFFERENT features from the ones training learned on. Live it
|
|
//--- was the same bug with a milder constant (the delta is ~1 bar, giving m_historyBars + 1).
|
|
//--- SWING_SCAN_CAP_BARS is the deepest lookback any feature performs (FindConfirmedZigZagPivot's
|
|
//--- bound); everything else in BufferTempDataCompute reaches less far.
|
|
int need = (int)m_historyBars + SWING_SCAN_CAP_BARS + MathMax(m_barrierHorizonBars, 1) + 2;
|
|
int barsNow = (int)MathMin(need, Bars(m_symbol.Name(), PERIOD_CURRENT));
|
|
if(!ResizeBuffers(barsNow) || !RefreshData())
|
|
return;
|
|
//--- INVALIDATE THE NOW-RELATIVE BAR CACHES. Non-obvious and load-bearing: the feature cache is keyed
|
|
//--- by MQL5 series index, and index 0 means "newest bar", so every closed candle shifts what every
|
|
//--- cached row stands for. Train() is the only other caller of this, and once m_trainingComplete is
|
|
//--- set ScheduleTrainingIfNeeded() routes every subsequent bar HERE instead - Train() is never
|
|
//--- re-entered, so without this call nothing ever clears the cache again for the rest of the process.
|
|
//--- A chart that trained to convergence (or was deployed via DeployNow()) would then keep replaying
|
|
//--- the rows computed for the last training era's bar grid: BufferTempData(0..m_historyBars-1) all hit
|
|
//--- the cache, the feature window never changes, and dPrevSignal freezes at its convergence-time value
|
|
//--- forever - silently, since every buffer above refreshed correctly and the vector is the right SHAPE.
|
|
//--- OnlineLearnStep() below would compound it by backpropping those stale features against freshly
|
|
//--- resolved labels, i.e. actively training the deployed model on mismatched pairs.
|
|
//--- A freshly started inference-only process (a backtest, or a buyer loading a deployed .nnw) was
|
|
//--- never affected: it never allocates these arrays, so BufferTempData()'s `cacheable` test is false
|
|
//--- and it always recomputes. This is a live/forward-chart fix, not a backtest one.
|
|
EnsureBarCachesCapacity(barsNow);
|
|
//--- Same bar grid, same panel. A deployed model never enters Train(), so this is the only place
|
|
//--- its cross-asset panel gets built - and it must be built from the SAME reference set training
|
|
//--- used, or inference reads a different feature vector than the weights were fitted to.
|
|
//--- Only as deep as inference actually reads. RefreshLatestSignal() touches bars 1..m_historyBars
|
|
//--- (window ends on the newest CLOSED bar - the +2 slack below covers the extra bar of depth)
|
|
//--- and the panel's own slow window reaches CROSSASSET_SLOW_BARS further back - nothing else. Asking
|
|
//--- for the full `barsNow` here would rebuild a training-depth panel on EVERY bar, which in the
|
|
//--- tester means one full multi-symbol resample per simulated bar. The cache check in
|
|
//--- BuildCrossAssetPanel is >=, so a deeper panel left over from training still satisfies this.
|
|
BuildCrossAssetPanel((int)m_historyBars + CROSSASSET_SLOW_BARS + 2);
|
|
EnsureSpreadSeries(barsNow);
|
|
//--- A deployed model never enters Train(), so this is the only place its barrier horizon gets
|
|
//--- measured - and OnlineLearnStep() below depends on it being right. First call sizes buffers
|
|
//--- against the fallback, which is harmless: `need` is dominated by SWING_SCAN_CAP_BARS either way.
|
|
EnsureBarrierHorizon(barsNow);
|
|
bool refreshed = RefreshLatestSignal();
|
|
//--- Continual learning: on a LIVE chart (never the tester/optimizer - OnlineLearnStep() self-guards
|
|
//--- on m_inferenceOnly) a deployed model keeps adapting to newly-confirmed structure. Runs AFTER the
|
|
//--- live signal is drawn (so the arrow uses the shadow as it was for THIS bar's decision) and BEFORE
|
|
//--- dtStudied advances (OnlineLearnStep keeps its own time watermark, independent of dtStudied).
|
|
OnlineLearnStep();
|
|
//--- Advance the live new-bar watermark ONLY on success. Advancing it unconditionally meant a
|
|
//--- transient window failure (indicator hole, history hiccup) closed the gate for the rest of the
|
|
//--- bar with the PREVIOUS bar's dPrevSignal still voting - the tester path (m_lastBarTime) already
|
|
//--- advanced only on success and self-healed; this is the live path catching up. On failure the
|
|
//--- gate stays open, so the next tick retries.
|
|
if(refreshed)
|
|
dtStudied = m_Time.GetData(0);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| |
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::RefreshLatestSignal(void)
|
|
{
|
|
//--- Meta target: no live inference path until S3 - see RefreshConvergedSignal's meta guard.
|
|
if(IsMetaTarget())
|
|
return false;
|
|
//--- Bar 1: the newest CLOSED bar, NOT the forming bar. This runs at the first tick after a bar
|
|
//--- opens, when series index 0 is a bar with one tick of data: (close-open)/atr ~ 0, high ~ low,
|
|
//--- a degenerate volume block, indicators computed on a 1-tick candle. Train() never produces
|
|
//--- such a window - every labeled bar is fully closed, and its label assumes entry at that bar's
|
|
//--- CLOSE (see TripleBarrierLabel's header). The training-parity query at this instant (fixed
|
|
//--- 2026-08-11) is therefore the window ending on bar 1, whose close IS the current price - the
|
|
//--- exact instant the label's hypothetical entry happens. The old i = 0 fed the deployed model an
|
|
//--- out-of-distribution final timestep - the timestep the LSTM/HYBRID output is keyed to - and
|
|
//--- semantically asked for the label of a bar whose close was still an hour away, so the deploy
|
|
//--- gate's OOS scores (closed bars, pass 3) measured a different query than live executed. Both
|
|
//--- paths go through BuildFeatureWindow(), which guarantees identical construction; this index is
|
|
//--- what makes them the same QUESTION.
|
|
int i = 1;
|
|
int r = i;
|
|
if(!BuildFeatureWindow(r))
|
|
{
|
|
//--- One combined failure now (partial window OR short total) where there used to be two counters.
|
|
//--- Kept distinct in the tally by testing what actually landed: a window that built every bar but
|
|
//--- came up short is the "short" case, anything else is a feature-build failure.
|
|
if(TempData.Total() > 0 && TempData.Total() < (int)m_historyBars * m_neuronsCount)
|
|
m_refreshFailShort++;
|
|
else
|
|
m_refreshFailFeatures++; // see PrintInferenceTally()
|
|
//--- No opinion this bar rather than a stale one: dPrevSignal still holds the PREVIOUS bar's
|
|
//--- decision, and LongCondition()/ShortCondition() would keep voting that stale direction all
|
|
//--- bar. The caller retries (RefreshConvergedSignal only advances dtStudied on success), so a
|
|
//--- transient failure costs ticks, not the bar.
|
|
dPrevSignal = 0.0;
|
|
return false;
|
|
}
|
|
//--- Live trading/inference reads from the EMA shadow net, not Net directly - see m_shadowNet's
|
|
//--- declaration comment. Falls back to Net if the shadow isn't bootstrapped yet (should only be
|
|
//--- momentarily, on a genuinely fresh start before EnsureShadowNet() has run).
|
|
EnsureShadowNet();
|
|
CNet *deployNet = (CheckPointer(m_shadowNet) != POINTER_INVALID) ? m_shadowNet : Net;
|
|
deployNet.feedForward(TempData);
|
|
deployNet.getResults(TempData);
|
|
if(m_outputNeuronsCount == 1)
|
|
dPrevSignal = TempData[0];
|
|
else
|
|
if(m_outputNeuronsCount == 3)
|
|
{
|
|
//--- Live decision. ApplyClassificationSoftmax() computes the softmax INTO TempData and returns
|
|
//--- the decision; AdjustedSignalFromSoftmax() re-reads that same TempData and applies the same
|
|
//--- strict-majority/ties-to-Neutral rule, so since the read-time prior correction was removed
|
|
//--- (2026-07-31) the two provably agree. The call is kept because a dozen sites name it as
|
|
//--- "the live decision rule" and that is still exactly what it is - the correction now lives
|
|
//--- in the trained weights instead of here.
|
|
//--- The "raw softmax was neutralized by prior correction" diagnostic that used to sit here went
|
|
//--- with it: with nothing between the two values it could never fire again.
|
|
ApplyClassificationSoftmax();
|
|
dPrevSignal = AdjustedSignalFromSoftmax();
|
|
}
|
|
m_refreshOk++;
|
|
//--- bt anchors the DECISION bar (bar 1, the closed bar the window ends on) - it keys the arrow,
|
|
//--- its High/Low placement and NMS declustering, and now matches the rescan path, which draws
|
|
//--- each historical arrow at the bar its window ends on.
|
|
datetime bt = m_Time.GetData(i);
|
|
//--- Keep a pure inference-side watermark of the newest bar FRAME this model has already evaluated.
|
|
//--- This must be the FORMING bar's open time (index 0), not bt: the new-bar gate compares it
|
|
//--- against SERIES_LASTBAR_DATE (also the forming bar's open), so anchoring it at bt (bar 1)
|
|
//--- would compare one bar behind and re-fire the refresh on every tick forever. The tester may
|
|
//--- load dtStudied from a live-chart save whose timestamp is AHEAD of the simulated backtest date
|
|
//--- range; using that training watermark to decide whether a "new bar" exists then freezes
|
|
//--- dPrevSignal at its init-bar value for the whole run. m_lastBarTime is this runtime's own
|
|
//--- latest evaluated frame instead, so it stays aligned to whichever history the current process
|
|
//--- is actually traversing.
|
|
m_lastBarTime = m_Time.GetData(0);
|
|
//--- LIVE NMS, AND IT NOW GATES THE TRADE, NOT JUST THE ARROW.
|
|
//---
|
|
//--- It used to sit at the bottom of this function wrapped around DrawObject() alone, so a suppressed
|
|
//--- bar lost its arrow and still traded: dPrevSignal was never touched, and dPrevSignal is what
|
|
//--- LongCondition()/ShortCondition()/SignedAIConfidence() read. The chart therefore showed roughly one
|
|
//--- arrow per EIGHT positions the EA would open - measured on SP500 H1 2026-08-09, where CONV called a
|
|
//--- direction on 64% of bars while ~40 arrows appeared across the ~500 visible ones. Worse, the arrows
|
|
//--- that survived were not a random eighth: rule 2 below keeps the HIGHER-CONFIDENCE side of a
|
|
//--- cluster, so the visible set was systematically the best member of each run. A chart that shows the
|
|
//--- best of every eight decisions and hides the rest reads far better than the model is, which is the
|
|
//--- same best-of-N selection error this codebase has now corrected in four other places - this time on
|
|
//--- the display layer, where it is most likely to mislead the person deciding whether to trade it.
|
|
//---
|
|
//--- Neutralising dPrevSignal (rather than adding a separate "may trade" flag consulted at each of the
|
|
//--- half-dozen read sites) is deliberate: it leaves exactly ONE definition of what this model decided
|
|
//--- this bar, so the arrow, the panel's "Current signal", the confidence handed to sizing/SL/TP/
|
|
//--- trailing, the refresh tally below and the order itself cannot drift apart again. One arrow is now
|
|
//--- one trade, which is what makes the chart an honest record.
|
|
//---
|
|
//--- NOTE the scoring consequence, deliberately NOT papered over: the era line's dir-precision still
|
|
//--- counts EVERY directional call, so it now describes a larger population than the one that trades.
|
|
//--- The era line carries a separate declustered figure alongside it (see m_oosNmsFired) so both are
|
|
//--- visible; the selection metric is not switched over until those numbers show what the coverage
|
|
//--- floor should be, because a blind switch is how the minRR and recall-floor catch-22s happened.
|
|
ENUM_SIGNAL lsig = DoubleToSignal(dPrevSignal);
|
|
bool nmsAccept = (lsig != Neutral) && NmsLiveAccept(bt, lsig, MathAbs(dPrevSignal));
|
|
if(lsig != Neutral && !nmsAccept)
|
|
dPrevSignal = 0.0; // declustered away: no arrow, no vote, no position
|
|
switch(DoubleToSignal(dPrevSignal))
|
|
{
|
|
case Buy:
|
|
m_refreshBuy++;
|
|
break;
|
|
case Sell:
|
|
m_refreshSell++;
|
|
break;
|
|
default:
|
|
m_refreshNeutral++;
|
|
break;
|
|
}
|
|
if(nmsAccept)
|
|
DrawObject(bt, dPrevSignal, m_High.GetData(i), m_Low.GetData(i));
|
|
else
|
|
DeleteObject(bt);
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| |
|
|
//+------------------------------------------------------------------+
|
|
double CExpertSignalAIBase::ApplyClassificationSoftmax(void)
|
|
{
|
|
// A non-finite logit poisons everything downstream: maxLogit, every exp(), the sum, and all three
|
|
// probabilities become NaN, and since NaN fails every comparison the two directional tests below
|
|
// are both false - so a NaN'd net returns Neutral on every bar forever and looks EXACTLY like a
|
|
// model that has simply gone quiet. That is the failure mode this project has chased repeatedly
|
|
// from the outside (panel says "no directional calls", nobody can tell whether the model is
|
|
// cautious or dead). Detect it here, at the one place the raw logits are first read, and say so.
|
|
if(!MathIsValidNumber(TempData.At(0)) || !MathIsValidNumber(TempData.At(1)) || !MathIsValidNumber(TempData.At(2)))
|
|
{
|
|
static int nanLogitReports = 0;
|
|
// Bounded: this cannot heal on its own (the weights are already corrupt), so unlimited logging
|
|
// would fill the journal for as long as the chart stays attached. Three is enough to prove it.
|
|
if(nanLogitReports < 3)
|
|
{
|
|
nanLogitReports++;
|
|
PrintFormat("%s: %s NON-FINITE network output (%g / %g / %g) - forcing Neutral. The weights are "
|
|
"corrupt; reload the last good .nnw or reset and retrain. Report %d of 3.",
|
|
__FUNCTION__, ID, TempData.At(0), TempData.At(1), TempData.At(2), nanLogitReports);
|
|
}
|
|
return 0;
|
|
}
|
|
// CLASS_LOGIT_SCALE (AI\Network.mqh) must match the training-gradient softmax in
|
|
// backProp/backPropOCL exactly - this is the same normalization the loss was trained against.
|
|
double maxLogit = CLASS_LOGIT_SCALE * MathMax(TempData.At(0), MathMax(TempData.At(1), TempData.At(2)));
|
|
double sum = 0;
|
|
for(int res = 0; res < 3; res++)
|
|
{
|
|
double temp = exp(CLASS_LOGIT_SCALE * TempData.At(res) - maxLogit);
|
|
sum += temp;
|
|
TempData.Update(res, temp);
|
|
}
|
|
for(int res = 0; res < 3; res++)
|
|
TempData.Update(res, TempData.At(res) / sum);
|
|
double pBuy = TempData.At(0);
|
|
double pSell = TempData.At(1);
|
|
double pNeutral = TempData.At(2);
|
|
// TempData.Maximum(0,3) scans left-to-right and keeps the FIRST index on a tie, so any tie
|
|
// (including the degenerate all-equal 0.3333/0.3333/0.3333 case from a collapsed/untrained net)
|
|
// always resolved to Buy (index 0) - silently turning "the model has no idea" into a directional
|
|
// trade. Buy/Sell now only win with a strict majority over BOTH other classes; every tie,
|
|
// 2-way or 3-way, falls through to Neutral.
|
|
if(pBuy > pSell && pBuy > pNeutral)
|
|
return pBuy; // Buy signal
|
|
if(pSell > pBuy && pSell > pNeutral)
|
|
return -pSell; // Sell signal
|
|
return 0; // Neutral signal (also the fallback on any tie)
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Post-hoc logit adjustment (prior correction) of the 3-class |
|
|
//| decision. Reads the raw softmax probabilities ApplyClassification|
|
|
//| Softmax() left in TempData[0..2] and returns the prior-corrected |
|
|
//| signed decision (+P'(buy)/-P'(sell)/0-neutral), the exact rule |
|
|
//| live trading fires on and the live-fired precision metric scores. |
|
|
//| |
|
|
//| RAW ARGMAX, deliberately. The prior correction this function used |
|
|
//| to apply at read time (Saerens et al. 2002) was REMOVED |
|
|
//| 2026-07-31 along with the AILogitPriorStrength input. |
|
|
//| |
|
|
//| Why there is nothing to correct here: the logit-adjusted loss |
|
|
//| adds tau*log(prior_c) to each class logit inside the TRAINING |
|
|
//| gradient, so the network learns to absorb the offset and its raw |
|
|
//| argmax is ALREADY the balanced-error-optimal decision. Applying a |
|
|
//| second correction at inference would account for the same base |
|
|
//| rate twice and push the decision back toward Neutral - undoing |
|
|
//| exactly what the loss bought. The old code knew this: the whole |
|
|
//| adjustment sat behind an `if(m_useLogitAdjustedLoss) return raw` |
|
|
//| guard and had been unreachable for the entire shipped default |
|
|
//| configuration. Kept as a named function rather than inlined |
|
|
//| because a dozen call sites document themselves by calling "the |
|
|
//| live decision rule" - and that is exactly what this is. |
|
|
//+------------------------------------------------------------------+
|
|
double CExpertSignalAIBase::AdjustedSignalFromSoftmax(void)
|
|
{
|
|
if(TempData.Total() < 3)
|
|
return 0.0;
|
|
double pBuy = TempData.At(0), pSell = TempData.At(1), pNeutral = TempData.At(2);
|
|
//--- Strict majority, ties to Neutral - the same rule as ApplyClassificationSoftmax(). The returned
|
|
//--- magnitude is a genuine probability, which the confidence floor and ConfidenceTier() read.
|
|
bool wantBuy = (pBuy > pSell && pBuy > pNeutral);
|
|
bool wantSell = (pSell > pBuy && pSell > pNeutral);
|
|
if(!wantBuy && !wantSell)
|
|
return 0.0;
|
|
//--- OPERATING POINT (2026-08-09). Argmax alone answers "which class is most likely"; it does not
|
|
//--- answer "is this worth trading", and those are different questions whenever the top two classes
|
|
//--- are nearly tied. A marginal directional win over Neutral used to become a trade, which is the
|
|
//--- mechanical source of the model calling a direction on ~90% of bars. Below the fitted margin
|
|
//--- this abstains instead - and abstaining is not a loss of information, it is the model declining
|
|
//--- to act on a distinction it cannot make. See DIR_CONF_THRESHOLD_BINS for how the value is chosen.
|
|
//---
|
|
//--- Returning Neutral rather than exposing a separate "tradeable" flag is deliberate, and matches
|
|
//--- the same decision made for live NMS (see RefreshLatestSignal): one definition of what this model
|
|
//--- decided this bar, so the arrow, the panel, the confidence handed to sizing/SL/TP, the OOS score
|
|
//--- and the order itself cannot drift apart.
|
|
if(m_dirConfThreshold > 0.0)
|
|
{
|
|
double win = wantBuy ? pBuy : pSell;
|
|
double rival = wantBuy ? MathMax(pSell, pNeutral) : MathMax(pBuy, pNeutral);
|
|
if((win - rival) < m_dirConfThreshold)
|
|
return 0.0;
|
|
}
|
|
return wantBuy ? pBuy : -pSell;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| The statistic the operating point is expressed in - see the |
|
|
//| declaration. Reads the softmax ALREADY in TempData, so callers |
|
|
//| must have run ApplyClassificationSoftmax() first. |
|
|
//+------------------------------------------------------------------+
|
|
double CExpertSignalAIBase::DirectionalMargin(void)
|
|
{
|
|
if(TempData.Total() < 3)
|
|
return -1.0;
|
|
double pBuy = TempData.At(0), pSell = TempData.At(1), pNeutral = TempData.At(2);
|
|
if(pBuy > pSell && pBuy > pNeutral)
|
|
return pBuy - MathMax(pSell, pNeutral);
|
|
if(pSell > pBuy && pSell > pNeutral)
|
|
return pSell - MathMax(pBuy, pNeutral);
|
|
return -1.0; // Neutral won: no directional call, so no operating point applies
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Clear the margin histogram at the start of the calibration walk. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ResetDirConfHistogram(void)
|
|
{
|
|
ArrayInitialize(m_dirConfBinCalls, 0);
|
|
ArrayInitialize(m_dirConfBinHits, 0);
|
|
m_dirConfPrimaryBars = 0;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| One calibration sample. isPrimaryBar survives from when this was |
|
|
//| harvested inside pass 2's oversampled replay queue, where counting |
|
|
//| duplicated minority bars would have fitted the operating point to |
|
|
//| a class balance the live model never sees (the same correction |
|
|
//| m_cumIsTotal makes - see its note in Training.mqh). The calibration |
|
|
//| walk visits each bar exactly once and passes true; the parameter |
|
|
//| stays so any future caller must state which it is. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::AccumulateDirConfSample(double margin, bool wasCorrect, bool isPrimaryBar)
|
|
{
|
|
if(!isPrimaryBar)
|
|
return;
|
|
//--- Counted BEFORE the directional test: this is the coverage denominator, so it has to be every
|
|
//--- primary bar the model scored, including the ones it called Neutral. Using only directional
|
|
//--- bars would make coverage 100% by construction at every threshold.
|
|
m_dirConfPrimaryBars++;
|
|
if(margin < 0.0)
|
|
return; // Neutral won - not a directional call
|
|
int bin = (int)(margin * DIR_CONF_THRESHOLD_BINS);
|
|
if(bin < 0)
|
|
bin = 0;
|
|
if(bin >= DIR_CONF_THRESHOLD_BINS)
|
|
bin = DIR_CONF_THRESHOLD_BINS - 1; // margin can reach exactly 1.0
|
|
m_dirConfBinCalls[bin]++;
|
|
if(wasCorrect)
|
|
m_dirConfBinHits[bin]++;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Choose the operating point: the margin that maximises EXPECTANCY |
|
|
//| on the held-out calibration slice while still calling a direction |
|
|
//| often enough to clear the SAME coverage floor the deploy gate |
|
|
//| uses. Held-out matters as much as the objective does - see |
|
|
//| DIR_CONF_CALIB_PCT_OF_IS for what fitting it on the training |
|
|
//| bars did to the sign of (p - break-even). |
|
|
//| |
|
|
//| Swept from the top down so the running totals are "calls at or |
|
|
//| above this bin", which is exactly the set a threshold there would |
|
|
//| admit - one pass, no nested loop over candidate thresholds. |
|
|
//| |
|
|
//| TIES GO TO THE LOWER THRESHOLD. Precision is a ratio of counts |
|
|
//| and plateaus over ranges of margin; taking the highest threshold |
|
|
//| on a plateau would buy identical precision for strictly less |
|
|
//| coverage, and coverage is what keeps the model tradeable. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::FitDirConfThreshold(void)
|
|
{
|
|
long totalCalls = 0;
|
|
for(int b = 0; b < DIR_CONF_THRESHOLD_BINS; b++)
|
|
totalCalls += m_dirConfBinCalls[b];
|
|
if(totalCalls < DIR_CONF_MIN_FIT_CALLS || m_dirConfPrimaryBars <= 0)
|
|
{
|
|
//--- Not enough evidence to place an operating point. KEEP THE PREVIOUS ONE - the old behaviour
|
|
//--- here was to reset to 0.0, which is "call a direction on every bar", the single most exposed
|
|
//--- setting in the range. A failed measurement must never decay to the most aggressive value it
|
|
//--- could have returned; the last threshold that WAS fitted is a strictly better estimate than
|
|
//--- the one setting we know maximises exposure. At era 0 the previous value is 0.0 regardless,
|
|
//--- so the cold-start path is unchanged.
|
|
if(!m_dirConfSparseWarned)
|
|
{
|
|
m_dirConfSparseWarned = true;
|
|
Print(ID + StringFormat(": directional confidence threshold NOT refitted - only %d directional "
|
|
"calls in the held-out calibration slice this era (need %d). Keeping "
|
|
"the previous operating point %.2f; this is normal for the first eras "
|
|
"and self-corrects as the model starts calling directions.",
|
|
(int)totalCalls, DIR_CONF_MIN_FIT_CALLS, m_dirConfThreshold));
|
|
}
|
|
return;
|
|
}
|
|
//--- The floor is the true directional base rate x MIN_COVERAGE_FRACTION_OF_BASE_RATE, matching
|
|
//--- Train()'s minCoveragePct exactly. Derived from THIS era's own IS labels rather than passed in,
|
|
//--- so the two cannot fall out of step when one of them is edited.
|
|
long trueDir = m_trueBuyCount + m_trueSellCount;
|
|
long trueTot = trueDir + m_trueNeutralCount;
|
|
double baseRatePct = (trueTot > 0) ? 100.0 * (double)trueDir / trueTot : 0.0;
|
|
double minCoveragePct = baseRatePct * MIN_COVERAGE_FRACTION_OF_BASE_RATE;
|
|
//--- EXPECTANCY, NOT PRECISION. Maximising the win rate alone has no answer for a PLATEAU, and the
|
|
//--- previous `precPct >= bestPrec` resolved one by walking to ever more coverage. That is a
|
|
//--- catastrophe on exactly the models that need a threshold most: a net with no edge scores its base
|
|
//--- rate at EVERY threshold, which is a perfect plateau, so the walk ran to bin 0 and returned
|
|
//--- threshold 0.0 - fire on every bar. Observed 2026-08-10 as PAI "overshooting signals" while the
|
|
//--- other three stayed selective; PAI has the most degenerate margin distribution (OOS outputs
|
|
//--- spanning the full 0.000..1.000 where CONV sits at 0.214..0.814), so its plateau is the flattest.
|
|
//---
|
|
//--- The money quantity is expectancy per BAR, and for a k:m barrier
|
|
//--- EV = (p - p0) * (k + m) with p0 = m/(m+k),
|
|
//--- so EV per bar = coverage * (p - p0) * (k + m). (k+m) is constant across thresholds, which
|
|
//--- leaves coverage * (p - p0) as the objective. It behaves correctly in all three regimes and
|
|
//--- needs no tie-break rule:
|
|
//--- p > p0 everywhere -> more coverage is more money, so it takes the coverage (the old
|
|
//--- behaviour, but for a reason rather than as a plateau artifact)
|
|
//--- p flat AT p0 -> every point scores 0 and the floor decides; no runaway
|
|
//--- p < p0 everywhere -> the LEAST coverage loses the least, so it becomes MORE selective
|
|
//--- instead of trading everything, which is the current reality for all
|
|
//--- four models and the opposite of what the old rule did.
|
|
double slMultFit, tpMultFit;
|
|
BarrierMultiples(slMultFit, tpMultFit);
|
|
double breakEvenPct = (slMultFit + tpMultFit > 0.0)
|
|
? 100.0 * slMultFit / (slMultFit + tpMultFit) : 50.0;
|
|
long runCalls = 0, runHits = 0;
|
|
double bestScore = -DBL_MAX;
|
|
double bestPrec = -1.0, bestThresh = 0.0, bestCov = 0.0;
|
|
for(int b = DIR_CONF_THRESHOLD_BINS - 1; b >= 0; b--)
|
|
{
|
|
runCalls += m_dirConfBinCalls[b];
|
|
runHits += m_dirConfBinHits[b];
|
|
if(runCalls <= 0)
|
|
continue;
|
|
double coveragePct = 100.0 * (double)runCalls / m_dirConfPrimaryBars;
|
|
if(coveragePct < minCoveragePct)
|
|
continue; // too selective to be deployable
|
|
double precPct = 100.0 * (double)runHits / runCalls;
|
|
double score = coveragePct * (precPct - breakEvenPct);
|
|
//--- Strict >, so a genuine tie keeps the MORE selective point (the loop reaches it first). The
|
|
//--- old >= did the reverse and that is what made the plateau run away.
|
|
if(score > bestScore)
|
|
{
|
|
bestScore = score;
|
|
bestPrec = precPct;
|
|
bestCov = coveragePct;
|
|
bestThresh = (double)b / DIR_CONF_THRESHOLD_BINS;
|
|
}
|
|
}
|
|
if(bestPrec < 0.0)
|
|
{
|
|
//--- Even calling on every directional argmax does not reach the coverage floor, so there is no
|
|
//--- room to be MORE selective. Unthresholded is then the only setting that can clear the gate.
|
|
m_dirConfThreshold = 0.0;
|
|
return;
|
|
}
|
|
double prevThresh = m_dirConfThreshold;
|
|
m_dirConfThreshold = bestThresh;
|
|
//--- Logged only when it actually moves a bin, so a stable operating point stays quiet.
|
|
if(MathAbs(m_dirConfThreshold - prevThresh) >= 1.0 / DIR_CONF_THRESHOLD_BINS)
|
|
Print(ID + StringFormat(": directional confidence threshold %.2f -> %.2f (fitted on %d HELD-OUT "
|
|
"calibration bars: %.1f%% coverage at %.1f%% WIN RATE vs " + DoubleToString(breakEvenPct, 1) +
|
|
"%% break-even, edge " + DoubleToString(bestPrec - breakEvenPct, 1) +
|
|
"pp, coverage floor %.1f%%). Below "
|
|
"this winner-vs-rival margin the model abstains instead of trading. The "
|
|
"rate is wins - target before stop on the side actually called - not "
|
|
"agreement with the collapsed 3-class label; see m_oosBuyPredictedWins.",
|
|
prevThresh, m_dirConfThreshold, (int)m_dirConfPrimaryBars, bestCov,
|
|
bestPrec, minCoveragePct));
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| EMA-updates the persisted true class base rates from a finished |
|
|
//| era's true class counts. First real measurement seeds directly; |
|
|
//| thereafter blended with the same smoothing as the accuracy/ |
|
|
//| confidence EMAs so one noisy era can't swing the live decision. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::UpdateClassPriors(long buyCnt, long sellCnt, long neutralCnt)
|
|
{
|
|
long tot = buyCnt + sellCnt + neutralCnt;
|
|
if(tot <= 0)
|
|
return;
|
|
double pb = (double)buyCnt / tot, ps = (double)sellCnt / tot, pn = (double)neutralCnt / tot;
|
|
if(m_priorNeutral <= 0.0) // first real measurement
|
|
{
|
|
m_priorBuy = pb;
|
|
m_priorSell = ps;
|
|
m_priorNeutral = pn;
|
|
return;
|
|
}
|
|
//--- (Was `m_useStaticPrior || m_freezePriorCalibration`. Those were two separate user-facing inputs
|
|
//--- whose only effect anywhere in the codebase was this one OR - two controls for one decision.
|
|
//--- UseStaticPrior was removed 2026-07-31; see the class-imbalance audit in Variables\Inputs.mqh.)
|
|
if(m_freezePriorCalibration)
|
|
return;
|
|
double k = Net.recentAverageSmoothingFactor;
|
|
if(k < 1.0)
|
|
k = 1.0;
|
|
m_priorBuy += (pb - m_priorBuy) / k;
|
|
m_priorSell += (ps - m_priorSell) / k;
|
|
m_priorNeutral += (pn - m_priorNeutral) / k;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Installs the training-time logit offsets - see the declaration. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ApplyLogitAdjustment(void)
|
|
{
|
|
if(CheckPointer(Net) == POINTER_INVALID)
|
|
return;
|
|
if(m_logitAdjustTau <= 0.0)
|
|
{
|
|
//--- Clear rather than merely skip: the input can be turned off on a chart that already installed
|
|
//--- offsets this session, and a stale adjustment would keep biasing the gradient silently.
|
|
Net.ClearLogitAdjustment();
|
|
return;
|
|
}
|
|
//--- Priors not measured yet (era 0 before the first tally, or a model with no .stats): leave the
|
|
//--- gradient unadjusted rather than guessing a distribution. The next era installs them.
|
|
//--- THIS USED TO BE SILENT, and that silence hid a whole-run failure: while the auto-tune search ran,
|
|
//--- UpdateClassPriors() was skipped in eval mode, so this branch was taken on EVERY era and the
|
|
//--- imbalance correction never once ran - with nothing in the log to say so. A mechanism that
|
|
//--- declines to act must announce it; the alternative is indistinguishable from working. Third time
|
|
//--- this codebase has been bitten by a quiet no-op, so it now warns every time it is not merely the
|
|
//--- expected era-0 case.
|
|
if(m_priorBuy <= 0.0 || m_priorSell <= 0.0 || m_priorNeutral <= 0.0)
|
|
{
|
|
if(m_eraCount > 0 && !m_logitAdjustSkipWarned)
|
|
{
|
|
m_logitAdjustSkipWarned = true;
|
|
Print(ID + ": WARNING - class-imbalance correction is NOT running at era " +
|
|
IntegerToString(m_eraCount) + ": the class priors have never been measured (Buy " +
|
|
DoubleToString(m_priorBuy, 4) + " Sell " + DoubleToString(m_priorSell, 4) + " Neutral " +
|
|
DoubleToString(m_priorNeutral, 4) + "). Training is falling back to plain cross-entropy, "
|
|
"which on a skewed label set collapses to the majority class.");
|
|
}
|
|
Net.ClearLogitAdjustment();
|
|
return;
|
|
}
|
|
//--- Effective tau, capped so the offsets cannot swamp the head's usable logit range - see
|
|
//--- LOGIT_ADJUST_MAX_RANGE_FRACTION. The binding quantity is the SPREAD between the largest and
|
|
//--- smallest offset, not their absolute size: softmax is shift-invariant, so a constant added to
|
|
//--- all three classes changes nothing and only their differences move the decision.
|
|
double lb = MathLog(m_priorBuy), ls = MathLog(m_priorSell), lnn = MathLog(m_priorNeutral);
|
|
double spread = MathMax(lb, MathMax(ls, lnn)) - MathMin(lb, MathMin(ls, lnn));
|
|
double tauEff = m_logitAdjustTau;
|
|
if(spread > 0.0)
|
|
{
|
|
double cap = LOGIT_ADJUST_MAX_RANGE_FRACTION * CLASS_LOGIT_SCALE / spread;
|
|
if(tauEff > cap)
|
|
tauEff = cap;
|
|
}
|
|
if(!m_logitAdjustLogged)
|
|
{
|
|
m_logitAdjustLogged = true;
|
|
Print(ID + ": logit adjustment - measured priors Buy " + DoubleToString(m_priorBuy * 100.0, 2) +
|
|
"% Sell " + DoubleToString(m_priorSell * 100.0, 2) + "% Neutral " +
|
|
DoubleToString(m_priorNeutral * 100.0, 2) + "% | log-prior spread " +
|
|
DoubleToString(spread, 2) + " against a logit range of " +
|
|
DoubleToString(CLASS_LOGIT_SCALE, 1) + " | tau " + DoubleToString(m_logitAdjustTau, 2) +
|
|
(tauEff < m_logitAdjustTau
|
|
? " CAPPED to " + DoubleToString(tauEff, 2) + " (uncapped it would consume " +
|
|
DoubleToString(100.0 * spread * m_logitAdjustTau / CLASS_LOGIT_SCALE, 0) +
|
|
"% of the range and saturate the head)"
|
|
: " (uncapped - within budget)"));
|
|
}
|
|
//--- ORDERED to match the output layer: [0]=Buy, [1]=Sell, [2]=Neutral - the order
|
|
//--- BuildFreshTopology emits and the order the softmax gradient reads (AI\Network.mqh).
|
|
double offsets[3];
|
|
offsets[0] = tauEff * lb;
|
|
offsets[1] = tauEff * ls;
|
|
offsets[2] = tauEff * lnn;
|
|
Net.SetLogitAdjustment(offsets);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Converts a double to ENUM_SIGNAL. |
|
|
//| 3-output (softmax classification) case: dPrevSignal's *sign* |
|
|
//| alone already encodes the argmax-selected class (+prob for Buy, |
|
|
//| -prob for Sell, exactly 0.0 for Neutral - see Train()/ |
|
|
//| RefreshLatestSignal()), so classification here is pure argmax: |
|
|
//| whichever class the network actually picked, full stop. No |
|
|
//| magnitude threshold is applied - confidence magnitude is a |
|
|
//| separate concern, already exposed via AIConfidence()/ |
|
|
//| SignedAIConfidence() (MathAbs(dPrevSignal)/dPrevSignal) for the |
|
|
//| signal engine's own confidence-weighted filters/lot sizing/SLTP, |
|
|
//| so this keeps "which class" and "how confident" decoupled. |
|
|
//| 1-output (tanh regression) case: unrelated network shape, keeps |
|
|
//| the original 0.50 magnitude cutoff as a genuine confidence gate. |
|
|
//+------------------------------------------------------------------+
|
|
ENUM_SIGNAL CExpertSignalAIBase::DoubleToSignal(double value)
|
|
{
|
|
value = NormalizeDouble(value, 2); // Round 'value' to two decimal places
|
|
if(value < -1.0 || value > 1.0)
|
|
return Undefine; // out of range, e.g. the -2 "not yet studied" sentinel
|
|
if(m_outputNeuronsCount == 3)
|
|
{
|
|
if(value > 0.0)
|
|
return Buy;
|
|
if(value < 0.0)
|
|
return Sell;
|
|
return Neutral;
|
|
}
|
|
if(value > 0.50)
|
|
return Buy;
|
|
if(value < -0.50)
|
|
return Sell;
|
|
return Neutral;
|
|
}
|
|
#endif // WARRIOR_AIBASE_INFERENCE_MQH
|