forked from animatedread/Warrior_EA
994fe3899c
The old target asked "which way is the next pivot", which every bar of a
~13-20 bar leg answers identically - so the net could not tell a fresh turn
from mid-trend and learned the prevailing direction instead. Its own
zero-skill reference showed it: chance sat at 56/44, i.e. the label WAS the
drift, and the gate's standing warning ("a model that only reproduces it has
found the drift, not an edge") applied to the target itself.
Buy now means a swing LOW commits within PIVOT_LABEL_TOLERANCE_BARS bars,
Sell a swing HIGH, Neutral no turn that close. Pivot type is read from
ZigZagBuffer[p] == Low[p], exact by construction in ZigZag.mq5. The existing
P1-final-once-P2-commits rule is kept and now also settles the NEGATIVE
verdict, so the Neutral majority is permanent rather than provisional.
Measured on a full fresh run, all 6 charts:
class balance 56/44/~0 -> 13.7/13.7/72.6 (imbalance 5.3:1)
label overlap ~31 bars -> 5 bars
independent obs 368-1086 -> 2331-7032
weights/obs 9.2-26.2 -> 1.1-4.2
coverage 100% of bars -> 17-48%
23 of 24 models fire all three classes at precision 18-32% vs 13-15%
chance; SP500's ensemble reaches DEPLOYABLE (32.3% vs a 24.0% bar).
Two bindings had to move with the label:
- The capacity deflator. m_swingLifespan fed EstimatedInSampleBars() as
raw/31, measured from the legs. Overlap is now a property of the LABEL -
one turn is callable by exactly the tolerance window - so it is the
window, not a leg measurement. Missing this would have kept every model
sized for a sixth of its real evidence.
- A dormant cold-start seed. Labels.mqh seeds the output bias toward the
dominant class above COLD_START_SEED_MIN_DOMINANCE (0.70); at 56/44 it
never armed, at 72.6% Neutral it does - writing a fixed +-3.0 against a
true prior spread of ~1.75, which would start every net predicting Neutral
~95% of the time. Now seeds the measured log-prior, zero-centred and
capped by the same guard rail the logit adjustment uses (Lin et al. 2017).
TGT:SWG1 -> TGT:PVT1:<tolerance>, with the window in the token because it is
part of the label: every .nnw is invalidated and the fleet retrains.
Depth is still gated, and now for a precise reason: the first dense layer
stays at FIRST_LAYER_MIN_WIDTH because budget = effN/(inputWidth+1) is 11.2
at input 624. Reaching the next rung needs inputWidth <= ~218, i.e. feature
pruning - not architecture.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
500 lines
30 KiB
MQL5
500 lines
30 KiB
MQL5
//+------------------------------------------------------------------+
|
|
//| Warrior_EA |
|
|
//| AnimateDread |
|
|
//| |
|
|
//| Swing-pivot labelling and the async label-cache prebuild. |
|
|
//+------------------------------------------------------------------+
|
|
#ifndef WARRIOR_AIBASE_LABELS_MQH
|
|
#define WARRIOR_AIBASE_LABELS_MQH
|
|
//+------------------------------------------------------------------+
|
|
//| A CLOSED CANDLE IS NOT A REASON TO RELABEL ANYTHING. |
|
|
//| |
|
|
//| Series indices are relative to now, so one new bar moves every |
|
|
//| cached bar's index by one. That used to invalidate the whole |
|
|
//| prebuild, which then rebuilt from scratch - on a timeframe where |
|
|
//| a bar closes faster than a run finishes, the labels were being |
|
|
//| recomputed continuously and the training set never held still. |
|
|
//| The labels themselves do not change: shift them and walk only the |
|
|
//| newest `delta` bars. |
|
|
//| |
|
|
//| Refuses (-> full rebuild) when a prebuild is mid-flight, since |
|
|
//| its cursor is an index into the array being moved. |
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::ShiftBarCaches(const int bars, const int delta)
|
|
{
|
|
if(delta <= 0 || bars <= delta || m_labelCacheBars <= 0 || m_labelPrebuildActive)
|
|
return false;
|
|
if(ArrayResize(m_labelCacheBuy, bars) < 0 || ArrayResize(m_labelCacheSell, bars) < 0
|
|
|| ArrayResize(m_labelResolveAge, bars) < 0 || ArrayResize(m_labelCacheHasValue, bars) < 0
|
|
|| ArrayResize(m_featureCacheHasValue, bars) < 0 || ArrayResize(m_featureCacheValid, bars) < 0
|
|
|| ArrayResize(m_featureCache, bars * m_neuronsCount) < 0)
|
|
return false;
|
|
//--- Backwards, so a source element is never overwritten before it is read.
|
|
for(int i = bars - 1; i >= delta; i--)
|
|
{
|
|
int j = i - delta;
|
|
m_labelCacheBuy[i] = m_labelCacheBuy[j];
|
|
m_labelCacheSell[i] = m_labelCacheSell[j];
|
|
m_labelResolveAge[i] = m_labelResolveAge[j];
|
|
m_labelCacheHasValue[i] = m_labelCacheHasValue[j];
|
|
m_featureCacheHasValue[i] = m_featureCacheHasValue[j];
|
|
m_featureCacheValid[i] = m_featureCacheValid[j];
|
|
int to = i * m_neuronsCount, from = j * m_neuronsCount;
|
|
for(int k = 0; k < m_neuronsCount; k++)
|
|
m_featureCache[to + k] = m_featureCache[from + k];
|
|
}
|
|
for(int i = 0; i < delta; i++)
|
|
{
|
|
m_labelCacheHasValue[i] = false;
|
|
m_featureCacheHasValue[i] = false;
|
|
}
|
|
m_labelCacheBars = bars;
|
|
m_labelCacheAnchorTime = m_Time.GetData(0);
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::EnsureBarCachesCapacity(int bars)
|
|
{
|
|
if(bars == m_labelCacheBars && m_Time.GetData(0) == m_labelCacheAnchorTime)
|
|
return false;
|
|
//--- New candles only: shift instead of wiping. Returns false = "nothing to rebuild", which is
|
|
//--- exactly what the caller does with an unchanged cache.
|
|
if(m_labelCacheAnchorTime > 0 && m_Time.GetData(0) > m_labelCacheAnchorTime
|
|
&& ShiftBarCaches(bars, bars - m_labelCacheBars))
|
|
return false;
|
|
ArrayResize(m_labelCacheBuy, bars);
|
|
ArrayResize(m_labelCacheSell, bars);
|
|
//--- Sized with the label caches they share a validity flag with, so they can never disagree
|
|
//--- about how many bars they cover.
|
|
ArrayResize(m_labelResolveAge, bars);
|
|
ArrayResize(m_labelCacheHasValue, bars);
|
|
ArrayInitialize(m_labelCacheHasValue, false);
|
|
ArrayResize(m_featureCache, bars * m_neuronsCount);
|
|
ArrayResize(m_featureCacheHasValue, bars);
|
|
ArrayResize(m_featureCacheValid, bars);
|
|
ArrayInitialize(m_featureCacheHasValue, false);
|
|
m_labelCacheBars = bars;
|
|
m_labelCacheAnchorTime = m_Time.GetData(0);
|
|
return true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Mean bars-to-resolution over the label cache. 1.0 until something |
|
|
//| has been measured, which makes EffectiveSampleSize() the identity |
|
|
//| - the pre-2026-08-17 behaviour. That default is deliberate: an |
|
|
//| UNMEASURED overlap must not silently shrink anyone's sample, so |
|
|
//| the correction switches itself on only once it has evidence. |
|
|
//+------------------------------------------------------------------+
|
|
double CExpertSignalAIBase::MeanLabelLifespan(void) const
|
|
{
|
|
//--- The cap is the label's structural bound: the scan window a resolution lag can never exceed.
|
|
return m_labelOverlap.MeanLifespan(SWING_SCAN_CAP_BARS);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Independent observations behind `rawN` overlapping labels. |
|
|
//+------------------------------------------------------------------+
|
|
double CExpertSignalAIBase::EffectiveSampleSize(double rawN) const
|
|
{
|
|
return m_labelOverlap.EffectiveSampleSize(rawN, SWING_SCAN_CAP_BARS);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Nearest confirmed ZigZag pivot at fromIdx or older (now-relative |
|
|
//| index, so "older" means scanning with INCREASING p - see this |
|
|
//| file's now-relative-index convention). |
|
|
//+------------------------------------------------------------------+
|
|
bool CExpertSignalAIBase::FindConfirmedZigZagPivot(int fromIdx, int &pivotIdx, double &pivotPrice, bool &pivotIsLow)
|
|
{
|
|
for(int p = MathMax(fromIdx, 0); p < fromIdx + SWING_SCAN_CAP_BARS; p++)
|
|
{
|
|
if(m_Open.GetData(p) == EMPTY_VALUE)
|
|
return false; // ran off the end of available history
|
|
double zz = m_zigZag.GetData(0, p);
|
|
if(zz == 0.0)
|
|
continue;
|
|
pivotIdx = p;
|
|
pivotPrice = zz;
|
|
pivotIsLow = (zz <= m_Low.GetData(p) + _Point);
|
|
return true;
|
|
}
|
|
return false;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| IS A SWING PIVOT ABOUT TO COMMIT, AND WHICH WAY DOES IT TURN? |
|
|
//| |
|
|
//| Buy = a swing LOW lands within PIVOT_LABEL_TOLERANCE_BARS |
|
|
//| bars of here - the turn up is at hand, buy it. |
|
|
//| Sell = a swing HIGH lands in that window - the turn down is. |
|
|
//| Neutral = no pivot that close. Most bars. Mid-leg is not a call. |
|
|
//| |
|
|
//| THIS REPLACED A DIRECTION-TO-NEXT-PIVOT LABEL (2026-08-25), and |
|
|
//| the distinction is the whole point. The old target asked "which |
|
|
//| side of the next pivot am I on", which every bar in a ~20-bar leg |
|
|
//| answers identically - so the net could not tell a fresh turn from |
|
|
//| mid-trend and simply learned the prevailing direction. Its own |
|
|
//| zero-skill reference showed it: chance sat at 56/44, i.e. the |
|
|
//| label WAS the drift, and the deploy gate's standing warning - |
|
|
//| "a model that only reproduces it has found the drift, not an |
|
|
//| edge" - applied to the target itself. This one fires only at the |
|
|
//| decision point, so a correct call is worth something. |
|
|
//| |
|
|
//| SECOND-ORDER, AND LARGE: label overlap collapses. Under the old |
|
|
//| target ~31 consecutive bars shared one pivot, so EffectiveSample- |
|
|
//| Size deflated 15,045 OOS calls to 440 independent ones and the |
|
|
//| deploy gate could not certify ANY edge for want of observations. |
|
|
//| Here a pivot marks only the PIVOT_LABEL_TOLERANCE_BARS bars that |
|
|
//| can call it - see m_lastLabelLifespan below. |
|
|
//| |
|
|
//| Geometry-free: the label owes nothing to a stop, a target or a |
|
|
//| horizon, which is what lets trade management be tuned separately |
|
|
//| instead of being baked into what the net learns. |
|
|
//| |
|
|
//| The pivot is m_zigZag's, the SAME definition the swing-context |
|
|
//| features already walk - one notion of "pivot" in the codebase, |
|
|
//| not two that can drift apart. |
|
|
//+------------------------------------------------------------------+
|
|
ENUM_SIGNAL CExpertSignalAIBase::SwingPivotDirectionLabel(int idx)
|
|
{
|
|
//--- LABEL-OVERLAP SPAN for the effective-sample machinery, NOT bars-to-resolution any more. What
|
|
//--- the SE correction needs is how many labelled bars share one underlying event, and one pivot
|
|
//--- can be called by exactly the PIVOT_LABEL_TOLERANCE_BARS bars that precede it. Resolution lag
|
|
//--- (how long until the pivot is CONFIRMED) is a different quantity and no longer belongs here:
|
|
//--- it says when a label may be trusted, not how much independent evidence it carries.
|
|
//--- 0 still means "unresolved", which is what gates the caching in AdvanceSwingLabelState.
|
|
m_lastLabelLifespan = 0;
|
|
double entry = m_Close.GetData(idx);
|
|
double atr = m_ATR.Main(idx);
|
|
//--- EMPTY_VALUE (a cold/short indicator read) IS DBL_MAX, and MathIsValidNumber(DBL_MAX) is
|
|
//--- true - it is a real finite number, just not one this indicator ever meant to report. Without
|
|
//--- the explicit == EMPTY_VALUE check, a cold ATR passes both tests, minMove below becomes
|
|
//--- ~1.8e307, and every bar in the sweep labels Neutral and is cached as resolved - permanently,
|
|
//--- since nothing currently invalidates the label cache when the indicator later warms up (see
|
|
//--- LabelCacheInvalidateAll()). FeatureBuilder.mqh's equivalent ATR guard already does this.
|
|
if(!MathIsValidNumber(entry) || entry <= 0.0 || entry == EMPTY_VALUE ||
|
|
!MathIsValidNumber(atr) || atr <= 0.0 || atr == EMPTY_VALUE)
|
|
return Neutral;
|
|
double spread = (double)m_symbol.Spread() * m_symbol.Point();
|
|
if(!MathIsValidNumber(spread) || spread < 0.0)
|
|
spread = 0.0;
|
|
double minMove = MathMax(2.0 * spread, 0.10 * atr);
|
|
//--- FINALITY IS AN EVENT, NOT A WAITING PERIOD. ZigZag.mq5's selection loop can only ever erase
|
|
//--- ZigZagBuffer[last_high_pos] while hunting a bottom, or [last_low_pos] while hunting a peak -
|
|
//--- so a pivot leaves the erasable slot for good the moment the OPPOSITE pivot is committed, and
|
|
//--- can never move again. P1 (the first pivot ahead of this bar) becomes final when P2 exists;
|
|
//--- pivots alternate by construction, so P2 is just the next non-zero bar and needs no type test.
|
|
//--- Until then the label is not knowable and the bar stays unresolved - that is the entire
|
|
//--- lookahead control for this target, exact rather than a confirmation-bar guess.
|
|
//---
|
|
//--- P1's finality also settles a NEGATIVE verdict, which is what makes the Neutral majority
|
|
//--- (~75% at the shipped tolerance) trustworthy rather than merely current: nothing can appear
|
|
//--- between here and P1, so if P1 is final and sits beyond the tolerance window, "no turn here"
|
|
//--- is permanent, not provisional. Without that, three quarters of the training set would be a
|
|
//--- label that could still change.
|
|
double p1Price = 0.0;
|
|
int p1Idx = -1;
|
|
for(int p = idx - 1; p >= MathMax(idx - SWING_SCAN_CAP_BARS, 1); p--)
|
|
{
|
|
if(m_Open.GetData(p) == EMPTY_VALUE)
|
|
return Neutral; // ran off loaded history before P2 confirmed
|
|
double pivot = m_zigZag.GetData(0, p);
|
|
//--- == EMPTY_VALUE, same reason as the atr/entry guard above: a cold ZigZag (buffer not yet
|
|
//--- filled) reads EMPTY_VALUE == DBL_MAX at every index, which is a real, positive,
|
|
//--- MathIsValidNumber()-passing number - so without this it reads as a pivot ABOVE every
|
|
//--- close, and every bar in the sweep labels Buy and is cached as resolved.
|
|
if(pivot == 0.0 || pivot == EMPTY_VALUE || !MathIsValidNumber(pivot))
|
|
continue;
|
|
if(p1Idx < 0)
|
|
{
|
|
p1Price = pivot;
|
|
p1Idx = p;
|
|
continue;
|
|
}
|
|
//--- P2 IS COMMITTED, SO P1 IS FINAL AND THIS BAR IS TRAINABLE. Everything below is decided.
|
|
//--- One pivot can be called by the PIVOT_LABEL_TOLERANCE_BARS bars in front of it, and that
|
|
//--- - not the distance to P2 - is what the effective-sample correction must divide by.
|
|
m_lastLabelLifespan = PIVOT_LABEL_TOLERANCE_BARS;
|
|
//--- OUT OF REACH: P1 is real and final but too far ahead to be this bar's call. Mid-leg.
|
|
if((idx - p1Idx) > PIVOT_LABEL_TOLERANCE_BARS)
|
|
return Neutral;
|
|
//--- IS THE LEG WORTH TAKING? The move that pays is the one AFTER the turn - P1 to P2 - not
|
|
//--- the approach to it. A ZigZag wiggle smaller than the spread is a pivot the indicator is
|
|
//--- entitled to draw and no one can trade, and labelling it Buy teaches the net to call
|
|
//--- turns that cost money to act on. Same minMove the old target charged, same reasoning.
|
|
if(MathAbs(p1Price - pivot) < minMove)
|
|
return Neutral;
|
|
//--- WHICH WAY IT TURNS. ZigZag.mq5 stores exactly High[p] at a peak or Low[p] at a bottom
|
|
//--- (ZigZag.mq5:140-165), so the comparison is a type test, not an approximation. Same
|
|
//--- idiom as FindConfirmedZigZagPivot() above, tolerance included.
|
|
bool p1IsLow = (p1Price <= m_Low.GetData(p1Idx) + _Point);
|
|
//--- A bottom ahead is a turn UP to be bought; a peak ahead is a turn DOWN to be sold. The
|
|
//--- PIVOT TYPE IS THE WHOLE SIGNAL - no comparison against this bar's close. Gating on
|
|
//--- whether the pivot sits above or below `entry` would drop exactly the bars where the turn
|
|
//--- has not finished coming to us, which is most of the early ones, and would bias the two
|
|
//--- classes asymmetrically the moment the leg is not symmetric around the close. How much
|
|
//--- adverse move is left before the turn is a trade-management question, and this label is
|
|
//--- deliberately geometry-free (see the header) so that stays tunable separately.
|
|
return (p1IsLow ? Buy : Sell);
|
|
}
|
|
return Neutral; // P1 still repainting, or no pivot pair inside the cap
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Resolves and caches the swing label for one bar. |
|
|
//| |
|
|
//| FINALITY-GATED CACHING: only a resolved label (P2 committed, so |
|
|
//| m_lastLabelLifespan > 0) may enter the cache. An unresolved bar |
|
|
//| is left uncached and revisited on a later pass - caching its |
|
|
//| provisional Neutral would freeze a label that is still unknowable |
|
|
//| and never update it once the pivot pair commits. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::AdvanceSwingLabelState(int idx, int bars)
|
|
{
|
|
if(idx < 2 || idx >= bars || m_labelCacheHasValue[idx])
|
|
return;
|
|
ENUM_SIGNAL verdict = SwingPivotDirectionLabel(idx);
|
|
if(m_lastLabelLifespan <= 0)
|
|
return;
|
|
//--- MEAN LABEL LIFESPAN, accumulated on the IS population only, matching the final tally pass:
|
|
//--- it deflates standard errors computed on that population, and a diagnostic that mixes two
|
|
//--- populations is worse than no diagnostic.
|
|
bool countable = (idx >= MathMax(2, m_labelPrebuildOosCutoff)
|
|
&& idx <= bars - MathMax(m_historyBars, 0) - 1);
|
|
if(countable)
|
|
m_labelOverlap.Accumulate(m_lastLabelLifespan);
|
|
m_labelCacheBuy[idx] = (verdict == Buy);
|
|
m_labelCacheSell[idx] = (verdict == Sell);
|
|
//--- Stored under the SAME validity flag as the label: the pool purge key reads it back as the
|
|
//--- earliest bar this label could have been known on.
|
|
if(idx < ArraySize(m_labelResolveAge))
|
|
m_labelResolveAge[idx] = m_lastLabelLifespan;
|
|
m_labelCacheHasValue[idx] = true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Kicks off the one-time eager label-cache pre-build for a fresh |
|
|
//| start (see m_labelCachePrebuilt's declaration comment). Computes |
|
|
//| the bar count/OOS split exactly as Train()'s era-start block |
|
|
//| would, then arms AdvanceLabelCachePrebuild() to do the actual |
|
|
//| chunked scan on this and subsequent Train() calls. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::StartLabelCachePrebuild(void)
|
|
{
|
|
//--- Not armed until the history is synced - the caller retries on its next scheduled call. Without
|
|
//--- this, a terminal restart ran the resumed-model pre-scan in the same second as OnInit, against
|
|
//--- whatever the terminal had loaded so far.
|
|
if(!SeriesInfoInteger(m_symbol.Name(), PERIOD_CURRENT, SERIES_SYNCHRONIZED))
|
|
return;
|
|
//--- ZIGZAG READINESS. A cold custom indicator (buffer not yet filled after Create()) reads
|
|
//--- EMPTY_VALUE at every index, including the newest bar - same signature ADIndicatorCold checks
|
|
//--- for feature reads (Expert\Features\FeatureBuilder.mqh). SwingPivotDirectionLabel() now refuses
|
|
//--- an EMPTY_VALUE pivot outright, but without this gate a cold sweep would just retry forever
|
|
//--- indistinguishably from "no pivot yet", and every already-scanned bar in that window still gets
|
|
//--- cached as resolved-Neutral by the ATR guard right above it in the same function. Checked
|
|
//--- directly rather than through ADIndicatorCold(): that helper also stamps the feature-builder's
|
|
//--- own transient-fail diagnostic (m_featureFailTransient/SetFailBlock), which belongs to a
|
|
//--- different call cycle (BufferTempDataCompute) and must not be touched from here.
|
|
if(m_zigZag.GetData(0, 0) == EMPTY_VALUE)
|
|
return; // retried on the next scheduled call, same contract as every guard in this function
|
|
//--- THE GAP IN THE TEARDOWN GUARDS (ad80e0b), found by the 2026-08-17 21:58 shutdown. Normally
|
|
//--- that is a once-per-run cost and it does not matter.
|
|
if(ShutdownRequested())
|
|
return;
|
|
//--- A model that is still TRAINING sizes its window by the training rule, not by the saved study
|
|
//--- watermark. Train()'s own era start applies this exact reset (TrainWindowStart) - this makes
|
|
//--- the pre-scan and the era loop agree. Deployed (complete) models keep their watermark: for them
|
|
//--- dtStudied gates INFERENCE recency, and this scan must not touch it.
|
|
if(!m_trainingComplete)
|
|
dtStudied = TrainWindowStart(m_tuneStartTrainBar);
|
|
int barsNow = (int)MathMin(Bars(m_symbol.Name(), PERIOD_CURRENT, dtStudied, TimeCurrent()) + m_historyBars, Bars(m_symbol.Name(), PERIOD_CURRENT));
|
|
//--- Clamped for TWO reasons, only one of which is about labels (see ServableBars()). So an
|
|
//--- unclamped prebuild here would re-break the very feature block Train()'s clamp just
|
|
//--- repaired, from a path that looks unrelated to it.
|
|
if(!ResizeBuffers(barsNow) || !RefreshData())
|
|
{
|
|
//--- NEVER SILENT AGAIN. MQL5's own "failed to get N bars" line was in the log the whole time
|
|
//--- and belonged to a stack frame nothing connected to the prebuild. Say which depth, and
|
|
//--- say it is fatal here.
|
|
if(!m_prebuildBlockWarned)
|
|
{
|
|
m_prebuildBlockWarned = true;
|
|
PrintFormat("%s: label prebuild BLOCKED - buffers would not prepare for %d bars. If MQL5"
|
|
" printed 'failed to get %d bars' just above, a buffer is being sized beyond the"
|
|
" %d bars this symbol actually has, and no era can start until that is fixed.",
|
|
ID, barsNow, barsNow, Bars(m_symbol.Name(), PERIOD_CURRENT));
|
|
}
|
|
return; // m_labelCachePrebuilt stays false, retried next call
|
|
}
|
|
int settled = SettledBars(barsNow, "label prebuild");
|
|
if(settled <= 0)
|
|
return; // depth still moving - retried next call, same contract as the line above
|
|
if(settled < barsNow)
|
|
{
|
|
barsNow = settled;
|
|
if(!ResizeBuffers(barsNow) || !RefreshData())
|
|
return;
|
|
}
|
|
EnsureBarCachesCapacity(barsNow);
|
|
int totalIter = (int)MathMax(barsNow - MathMax(m_historyBars, 0), 0);
|
|
m_labelPrebuildBars = barsNow;
|
|
m_labelPrebuildOosCutoff = (int)(MathMax(0, MathMin(100, m_oosSplitPct)) / 100.0 * totalIter);
|
|
m_labelPrebuildIndex = (int)(barsNow - MathMax(m_historyBars, 0) - 1);
|
|
m_labelPrebuildBuyCount = 0;
|
|
m_labelPrebuildSellCount = 0;
|
|
m_labelPrebuildNeutralCount = 0;
|
|
//--- Reset WITH the cache, not once per process: lifespans measured under an older window answer
|
|
//--- a different question, and carrying them forward would deflate the new standard errors by the
|
|
//--- old overlap.
|
|
m_labelOverlap.Reset();
|
|
m_labelPrebuildActive = true;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Advances the eager label-cache pre-build by up to a time budget, |
|
|
//| then yields (same chunking pattern as the era loop). Mirrors the |
|
|
//| era loop's own labeling eligibility gate (minus the dPrevSignal |
|
|
//| check, meaningless pre-first-feedForward). On completion, seeds |
|
|
//| m_prevEraTrueBuyCount/Sell/Neutral from the upfront IS-only tally |
|
|
//| so era 0's class priors are measured, not empty. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::AdvanceLabelCachePrebuild(void)
|
|
{
|
|
const uint PREBUILD_TIME_BUDGET_MS = 80;
|
|
uint chunkStartTick = GetTickCount();
|
|
int i;
|
|
for(i = m_labelPrebuildIndex; i >= 2; i--)
|
|
{
|
|
//--- Already chunked at 80 ms, so this costs at most one chunk - but the tally pass below is NOT
|
|
//--- chunked, and on a stop there is no reason to walk the rest of the window to reach it.
|
|
//--- Resumable by construction: m_labelPrebuildIndex is written before returning either way.
|
|
if(ShutdownRequested())
|
|
{
|
|
m_labelPrebuildIndex = i;
|
|
return;
|
|
}
|
|
if(GetTickCount() - chunkStartTick >= PREBUILD_TIME_BUDGET_MS)
|
|
{
|
|
m_labelPrebuildIndex = i;
|
|
return;
|
|
}
|
|
if(!(i < (int)(m_labelPrebuildBars - MathMax(m_historyBars, 0) - 1) && m_Time.GetData(i) > dtStudied))
|
|
continue;
|
|
if(!m_labelCacheHasValue[i])
|
|
AdvanceSwingLabelState(i, m_labelPrebuildBars);
|
|
}
|
|
//--- Final tally pass (IS-only, matches isOOS = (i < oosCutoff) used by the era loop).
|
|
for(i = m_labelPrebuildBars - MathMax(m_historyBars, 0) - 1; i >= MathMax(2, m_labelPrebuildOosCutoff); i--)
|
|
{
|
|
if(!m_labelCacheHasValue[i])
|
|
continue; // unresolved, or outside the dtStudied/window-edge eligibility gate above
|
|
if(m_labelCacheBuy[i])
|
|
m_labelPrebuildBuyCount++;
|
|
else
|
|
if(m_labelCacheSell[i])
|
|
m_labelPrebuildSellCount++;
|
|
else
|
|
m_labelPrebuildNeutralCount++;
|
|
}
|
|
//--- Prebuild complete - seed era 0's class base rates from the real upfront tally instead of leaving
|
|
//--- UpdateClassPriors() nothing to measure (see m_prevEraTrueBuyCount's declaration comment).
|
|
//--- Consumed (and cleared) by Train()'s era-start block on era 0 specifically - m_prebuildSeedPending.
|
|
m_prevEraTrueBuyCount = m_labelPrebuildBuyCount;
|
|
m_prevEraTrueSellCount = m_labelPrebuildSellCount;
|
|
m_prevEraTrueNeutralCount = m_labelPrebuildNeutralCount;
|
|
m_prebuildSeedPending = true;
|
|
m_labelCachePrebuilt = true;
|
|
m_labelPrebuildActive = false;
|
|
//--- Measured-imbalance visibility. A log line must describe what the code DID: report the
|
|
//--- measured distribution, which is real and useful, and nothing else.
|
|
int prebuildMinDir = (int)MathMin(m_labelPrebuildBuyCount, m_labelPrebuildSellCount);
|
|
int prebuildMaxCls = (int)MathMax(m_labelPrebuildNeutralCount, MathMax(m_labelPrebuildBuyCount, m_labelPrebuildSellCount));
|
|
string prebuildRatioInfo = (prebuildMinDir > 0 && prebuildMaxCls > 0)
|
|
? " | measured imbalance ~" + DoubleToString((double)prebuildMaxCls / prebuildMinDir, 1) + ":1"
|
|
: " | measured imbalance n/a (a directional class has no labeled bars in this window)";
|
|
int prebuildTotal = m_labelPrebuildBuyCount + m_labelPrebuildSellCount + m_labelPrebuildNeutralCount;
|
|
string prebuildShare = (prebuildTotal > 0)
|
|
? " | share Buy " + DoubleToString(100.0 * m_labelPrebuildBuyCount / prebuildTotal, 1) +
|
|
"% Sell " + DoubleToString(100.0 * m_labelPrebuildSellCount / prebuildTotal, 1) +
|
|
"% Neutral " + DoubleToString(100.0 * m_labelPrebuildNeutralCount / prebuildTotal, 1) + "%"
|
|
: "";
|
|
//--- LABEL OVERLAP, printed with the distribution because it is a property of the same
|
|
//--- measurement and because every standard error downstream is divided by it. Under the
|
|
//--- pivot-event target the overlap is the TOLERANCE WINDOW - the bars that can call one turn -
|
|
//--- not the resolution lag, which is a statement about when a label may be trusted rather than
|
|
//--- about how much independent evidence it carries. See SwingPivotDirectionLabel().
|
|
string prebuildOverlap = "";
|
|
if(m_labelOverlap.Count() > 0)
|
|
prebuildOverlap = StringFormat(" | label overlap %.1f bars (the window of bars that can call one pivot) -> "
|
|
"%d labels are worth ~%d independent ones (every SE below is sized on that)",
|
|
MeanLabelLifespan(), (int)m_labelOverlap.Count(),
|
|
(int)EffectiveSampleSize((double)m_labelOverlap.Count()));
|
|
Print(ID + ": label cache pre-built - IS true-label distribution -> Buy: " + IntegerToString(m_labelPrebuildBuyCount) +
|
|
" | Sell: " + IntegerToString(m_labelPrebuildSellCount) + " | Neutral: " + IntegerToString(m_labelPrebuildNeutralCount) +
|
|
prebuildRatioInfo + prebuildShare + prebuildOverlap +
|
|
(m_eraCount == 0 ? StringFormat(" (seeding era 0 - PIVOT-EVENT target: Buy/Sell mean 'a swing low/high commits"
|
|
" within %d bars', Neutral means no turn that close, which is most bars)",
|
|
(int)PIVOT_LABEL_TOLERANCE_BARS)
|
|
: " (mid-run rebuild after new-bar cache invalidation - era " + IntegerToString(m_eraCount) + " resumes on the relabeled window)"));
|
|
//--- Cold-start fix: a freshly-initialized (random-weight) network's argmax is close to uniform
|
|
//--- noise across the 3 classes, so on this typically heavily-skewed label distribution it fires
|
|
//--- far more non-majority-class calls at the very start of era 0 than the true base rate
|
|
//--- warrants, until enough backProp steps correct it.
|
|
if(m_outputNeuronsCount == 3 && m_eraCount == 0)
|
|
{
|
|
int dominant = 2; // Neutral
|
|
int dominantCount = m_labelPrebuildNeutralCount;
|
|
if(m_labelPrebuildBuyCount > dominantCount)
|
|
{
|
|
dominant = 0;
|
|
dominantCount = m_labelPrebuildBuyCount;
|
|
}
|
|
if(m_labelPrebuildSellCount > dominantCount)
|
|
{
|
|
dominant = 1;
|
|
dominantCount = m_labelPrebuildSellCount;
|
|
}
|
|
int totalLabeled = m_labelPrebuildBuyCount + m_labelPrebuildSellCount + m_labelPrebuildNeutralCount;
|
|
if(totalLabeled > 0 && (double)dominantCount / totalLabeled > COLD_START_SEED_MIN_DOMINANCE)
|
|
{
|
|
//--- SEED THE MEASURED LOG-PRIOR, NOT A FIXED MAGNITUDE. This used to write +-3.0
|
|
//--- (softmax ~0.95/0.05) toward whichever class dominated. That never executed while the
|
|
//--- target was direction-to-next-pivot - the split there was ~56/44/0, under
|
|
//--- COLD_START_SEED_MIN_DOMINANCE - and the pivot-event target is the first label to arm
|
|
//--- it, at ~12/12/75. A +-3 seed is a 6-logit spread against a TRUE prior spread of
|
|
//--- log(0.75/0.125) ~= 1.79: it would start the net predicting Neutral ~95% of the time,
|
|
//--- roughly 3.4x more skewed than the data, and the two rare directional classes then have
|
|
//--- to climb out of that on top of the residual imbalance the capped logit adjustment
|
|
//--- already leaves them (ApplyLogitAdjustment's tau pins the correction at 1.2 logits).
|
|
//--- That is the shape of the collapse this project already paid for once (1b5a412).
|
|
//---
|
|
//--- Initialising the output bias to the class log-prior is the standard prescription for
|
|
//--- exactly this rare-event regime (Lin et al. 2017, focal loss, sec. 4.1 "prior"): it
|
|
//--- makes the untrained net predict the base rate instead of uniform noise, which is what
|
|
//--- the original comment below wanted, without overshooting the base rate it is matching.
|
|
//--- Zero-centred because softmax is shift-invariant - only the differences are real.
|
|
double priors[3];
|
|
priors[0] = (double)m_labelPrebuildBuyCount / totalLabeled;
|
|
priors[1] = (double)m_labelPrebuildSellCount / totalLabeled;
|
|
priors[2] = (double)m_labelPrebuildNeutralCount / totalLabeled;
|
|
//--- Floor: an empty class must not seed a -inf bias. 1e-4 is well below any share that
|
|
//--- survives the MIN_OOS_CLASS_SAMPLES_FOR_GATE-scale counts this runs on.
|
|
double logs[3], mean = 0.0;
|
|
for(int c = 0; c < 3; c++)
|
|
{
|
|
logs[c] = MathLog(MathMax(priors[c], 1e-4));
|
|
mean += logs[c];
|
|
}
|
|
mean /= 3.0;
|
|
//--- Same guard rail the logit adjustment uses: never consume more than
|
|
//--- LOGIT_ADJUST_MAX_RANGE_FRACTION of the head's usable logit range.
|
|
double seedCap = LOGIT_ADJUST_MAX_RANGE_FRACTION * CLASS_LOGIT_SCALE;
|
|
double biasValues[3];
|
|
for(int c = 0; c < 3; c++)
|
|
biasValues[c] = MathMax(-seedCap, MathMin(seedCap, logs[c] - mean));
|
|
if(Net.SeedOutputLayerBias(biasValues))
|
|
PrintVerbose(ID + StringFormat(": seeded output layer bias to the measured class log-prior"
|
|
" B/S/N = %+.2f/%+.2f/%+.2f (shares %.1f%%/%.1f%%/%.1f%%, dominant %s at"
|
|
" %.1f%% > %.0f%% seed threshold) - era 0 cold-start fix, so the untrained"
|
|
" net starts at the base rate rather than at uniform noise.",
|
|
biasValues[0], biasValues[1], biasValues[2],
|
|
100.0 * priors[0], 100.0 * priors[1], 100.0 * priors[2],
|
|
EnumToString((ENUM_SIGNAL)(dominant == 0 ? Buy : dominant == 1 ? Sell : Neutral)),
|
|
100.0 * dominantCount / totalLabeled, 100.0 * COLD_START_SEED_MIN_DOMINANCE));
|
|
}
|
|
}
|
|
}
|
|
#endif // WARRIOR_AIBASE_LABELS_MQH
|