forked from animatedread/Warrior_EA
b461844767
TWO things, one incident. 1) THE BUG I SHIPPED IN0c85c54. m_tuneStartTrainBar is declared, initialised to 0, and NEVER ASSIGNED - the assignment existed before the God-class split and the split dropped it, leaving a dead member. Harmless while nothing read it; a real defect the moment0c85c54made StartLabelCachePrebuild() reset dtStudied from it. Train() then computed the window as max(StartTrainBar, floor) while the prebuild computed max(0, floor), where StartTrainBar is the non-zero datetime OnChartEventHandler passes through from the "New Bar" event. The two therefore disagreed about `bars`, so EnsureBarCachesCapacity() saw a changed size at era start, wiped the caches, and re-armed a full 38k-bar prebuild - instead of training. Restored the assignment so both sides evaluate the identical expression. 2) THE REASON IT TOOK ALL NIGHT TO FIND. Train() is a state machine with six early-return branches above the era loop and every one of them is silent. Four charts burned a core each for 15 minutes with an empty journal: the pass heartbeats (694b756) proved the era loop was never reached, no prebuild completion line appeared either, and nothing external can see inside a single MQL5 thread - per-thread CPU says "busy", file writes say nothing, and the VPS has no debugger. That is an undiagnosable state, and it is the thing to fix, not just the bug of the day. ReportTrainStall() now names the branch Train() is taking whenever no era has completed for 3 minutes, at most once a minute per signal, with the state that decides the branch: run/prebuild/simOos/resume flags, era, dtStudied, and - for the cache-invalidation branch specifically - BOTH bar counts, since two sizings disagreeing is exactly what re-arms the prebuild forever. Silent on a healthy run: an era completing resets the clock. Both build variants compile 0 errors, 0 warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1639 lines
98 KiB
MQL5
1639 lines
98 KiB
MQL5
//+------------------------------------------------------------------+
|
|
//| Warrior_EA |
|
|
//| AnimateDread |
|
|
//| |
|
|
//| Filter-based indicator auto-tuner (mutual information scoring). |
|
|
//| |
|
|
//| PARTIAL IMPLEMENTATION FILE - not standalone. |
|
|
//| This holds CExpertSignalAIBase method BODIES only. The class |
|
|
//| declaration lives in Expert\ExpertSignalAIBase.mqh, which |
|
|
//| #includes this file at the bottom, after the declaration. Do not |
|
|
//| include it anywhere else and do not compile it on its own. |
|
|
//| |
|
|
//| Split out purely to make the 8216-line original navigable; the |
|
|
//| code inside was moved verbatim, not rewritten. |
|
|
//+------------------------------------------------------------------+
|
|
#ifndef WARRIOR_AIBASE_AUTOTUNE_MQH
|
|
#define WARRIOR_AIBASE_AUTOTUNE_MQH
|
|
#ifdef WARRIOR_EXPORT_FEATURES
|
|
//+------------------------------------------------------------------+
|
|
//| RESEARCH BUILD ONLY - see the declaration comment. |
|
|
//| |
|
|
//| Every research question so far has cost a compile, a deploy, an |
|
|
//| attach and a log read - minutes each, and the answer arrives one |
|
|
//| hypothesis at a time. That loop, not the modelling, is what has |
|
|
//| made this slow. Exporting the feature matrix ONCE moves the whole |
|
|
//| question offline, where a hypothesis costs seconds and real tools |
|
|
//| (joint mutual information, gradient boosting, proper walk-forward |
|
|
//| cross-validation) are available - none of which can be written in |
|
|
//| MQL5 in reasonable time. |
|
|
//| |
|
|
//| Exports the RAW BARS next to the features deliberately: with OHLC |
|
|
//| and ATR offline, every barrier geometry, every horizon and every |
|
|
//| in-trade target can be recomputed without touching MetaTrader |
|
|
//| again. The bar TIME goes out too, which makes session, hour and |
|
|
//| day-of-week features derivable for free - and those are the only |
|
|
//| inputs in play that are NOT a transform of the same OHLCV series. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ExportFeatureMatrix(void)
|
|
{
|
|
if(MQLInfoInteger(MQL_OPTIMIZATION))
|
|
return;
|
|
int barsNow = Bars(m_symbol.Name(), PERIOD_CURRENT);
|
|
if(barsNow <= m_historyBars + 2)
|
|
{
|
|
Print(ID + ": EXPORT - only " + IntegerToString(barsNow) + " bars available, nothing to write");
|
|
return;
|
|
}
|
|
if(!ResizeBuffers(barsNow) || !RefreshData())
|
|
{
|
|
Print(ID + ": EXPORT - buffers not ready (" + IntegerToString(barsNow) + " bars), aborting");
|
|
return;
|
|
}
|
|
EnsureBarCachesCapacity(barsNow);
|
|
EnsureBarrierHorizon(barsNow);
|
|
string dir = eaName + "\\Research\\";
|
|
string fn = dir + m_symbol.Name() + "_" + IntegerToString(_Period) + "_features.csv";
|
|
int h = FileOpen(fn, FILE_COMMON | FILE_WRITE | FILE_CSV | FILE_ANSI, ',');
|
|
if(h == INVALID_HANDLE)
|
|
{
|
|
Print(ID + ": EXPORT - cannot open " + fn + ", error " + IntegerToString(GetLastError()));
|
|
return;
|
|
}
|
|
string header = "idx,time,open,high,low,close,atr";
|
|
for(int f = 0; f < m_neuronsCount; f++)
|
|
header += ",f" + IntegerToString(f);
|
|
FileWrite(h, header);
|
|
//--- Oldest first. The loop walks DOWN the series index, which is forward in time (higher index =
|
|
//--- older), so the file reads chronologically and Python can treat row order as time order.
|
|
int written = 0, skipped = 0;
|
|
uint t0 = GetTickCount();
|
|
for(int i = barsNow - 1; i >= 0; i--)
|
|
{
|
|
TempData.Clear();
|
|
if(!BufferTempData(i) || TempData.Total() < m_neuronsCount)
|
|
{
|
|
skipped++;
|
|
continue;
|
|
}
|
|
double atr = m_ATR.Main(i);
|
|
string row = IntegerToString(i) + "," + IntegerToString((long)m_Time.GetData(i)) + "," +
|
|
DoubleToString(m_Open.GetData(i), _Digits) + "," +
|
|
DoubleToString(m_High.GetData(i), _Digits) + "," +
|
|
DoubleToString(m_Low.GetData(i), _Digits) + "," +
|
|
DoubleToString(m_Close.GetData(i), _Digits) + "," +
|
|
DoubleToString(MathIsValidNumber(atr) ? atr : 0.0, _Digits);
|
|
for(int f = 0; f < m_neuronsCount; f++)
|
|
row += "," + DoubleToString(TempData.At(f), 8);
|
|
FileWrite(h, row);
|
|
written++;
|
|
}
|
|
TempData.Clear();
|
|
FileClose(h);
|
|
Print(ID + StringFormat(": EXPORT COMPLETE - %d rows x %d features -> Common\\Files\\%s "
|
|
"(%d bars skipped for missing features, %.1fs, horizon %d, spread %d points)",
|
|
written, m_neuronsCount, fn, skipped, (GetTickCount() - t0) / 1000.0,
|
|
m_barrierHorizonBars, (int)m_symbol.Spread()));
|
|
ExportRawRates();
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| RESEARCH BUILD ONLY. Raw OHLCV for a GRID of symbols/timeframes, |
|
|
//| not just this chart's. |
|
|
//| |
|
|
//| The 26 engineered features can only be produced for the chart the |
|
|
//| EA is attached to - the indicator handles are bound to |
|
|
//| PERIOD_CURRENT. Raw rates are not: CopyRates serves any symbol |
|
|
//| and any timeframe from a single chart. So one attach yields the |
|
|
//| whole research grid, and every question that does not require the |
|
|
//| EXISTING feature set - a different horizon, a different barrier, |
|
|
//| session/time-of-day effects, features this EA does not have yet - |
|
|
//| can then be answered offline without MetaTrader in the loop at |
|
|
//| all. That is what turns a per-hypothesis cost of minutes into |
|
|
//| seconds, which has been the real bottleneck all along. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ExportRawRates(void)
|
|
{
|
|
string symbols[] = { "SP500", "USDJPY", "XAUUSD", "EURUSD", "GBPUSD", "US30", "NAS100", "BTCUSD" };
|
|
ENUM_TIMEFRAMES tfs[] = { PERIOD_M5, PERIOD_M15, PERIOD_H1, PERIOD_H4, PERIOD_D1 };
|
|
string dir = eaName + "\\Research\\";
|
|
int cells = 0, rowsTotal = 0;
|
|
for(int s = 0; s < ArraySize(symbols); s++)
|
|
{
|
|
//--- Skip silently rather than warn: the grid is deliberately broader than any one broker's symbol
|
|
//--- list, so an absent instrument is expected, not an error.
|
|
if(!SymbolSelect(symbols[s], true))
|
|
continue;
|
|
for(int p = 0; p < ArraySize(tfs); p++)
|
|
{
|
|
MqlRates r[];
|
|
ArraySetAsSeries(r, false); // oldest first, so file order is time order
|
|
int got = CopyRates(symbols[s], tfs[p], 0, 200000, r);
|
|
if(got <= 100)
|
|
continue;
|
|
string fn = dir + symbols[s] + "_" + IntegerToString((int)tfs[p]) + "_rates.csv";
|
|
int h = FileOpen(fn, FILE_COMMON | FILE_WRITE | FILE_CSV | FILE_ANSI, ',');
|
|
if(h == INVALID_HANDLE)
|
|
continue;
|
|
int dg = (int)SymbolInfoInteger(symbols[s], SYMBOL_DIGITS);
|
|
FileWrite(h, "time,open,high,low,close,tickvol,spread");
|
|
for(int i = 0; i < got; i++)
|
|
FileWrite(h, IntegerToString((long)r[i].time) + "," +
|
|
DoubleToString(r[i].open, dg) + "," + DoubleToString(r[i].high, dg) + "," +
|
|
DoubleToString(r[i].low, dg) + "," + DoubleToString(r[i].close, dg) + "," +
|
|
IntegerToString((long)r[i].tick_volume) + "," + IntegerToString(r[i].spread));
|
|
FileClose(h);
|
|
cells++;
|
|
rowsTotal += got;
|
|
Print(ID + StringFormat(": EXPORT rates - %s %s: %d bars", symbols[s],
|
|
EnumToString(tfs[p]), got));
|
|
}
|
|
}
|
|
Print(ID + StringFormat(": EXPORT RATES COMPLETE - %d cells, %d bars total, under Common\\Files\\%s",
|
|
cells, rowsTotal, dir));
|
|
}
|
|
#endif
|
|
//--- The genetic + successive-halving helpers that used to live here (GaRungEras, GaExtract, GaStore,
|
|
//--- GaMutate, GaRandomCandidate, GaBlockCrossover, GaSortAliveByScoreDesc, GaBreedNextGeneration) were
|
|
//--- deleted on 2026-08-01 together with the search they served. See TuneIndicatorsByFilter() below for
|
|
//--- the measured cost that retired them and what replaced it.
|
|
//+------------------------------------------------------------------+
|
|
//| MUTUAL INFORMATION between one cached feature column and the |
|
|
//| triple-barrier label, in nats, over a sample of in-sample bars. |
|
|
//| |
|
|
//| I(X;Y) = sum p(x,y) log( p(x,y) / (p(x) p(y)) ), with the feature |
|
|
//| discretised into MI_BINS EQUAL-FREQUENCY bins. Equal-frequency |
|
|
//| rather than equal-width because these features are ATR-normalised |
|
|
//| and heavy-tailed: fixed-width bins put nearly everything in one |
|
|
//| bucket and report ~0 information for a genuinely useful feature. |
|
|
//| |
|
|
//| Rank-based binning gives equal frequency for free - sort a copy of |
|
|
//| the column, then a value's bin is its rank scaled into MI_BINS. |
|
|
//+------------------------------------------------------------------+
|
|
double CExpertSignalAIBase::FeatureColumnMI(const double &vals[], const int &labels[], int n)
|
|
{
|
|
if(n < MI_MIN_SAMPLES)
|
|
return 0.0;
|
|
double sorted[];
|
|
ArrayResize(sorted, n);
|
|
ArrayCopy(sorted, vals, 0, 0, n);
|
|
ArraySort(sorted);
|
|
//--- A column that never varies carries no information; short-circuit so the log below is never
|
|
//--- reached with a degenerate single-bin histogram.
|
|
if(sorted[0] == sorted[n - 1])
|
|
return 0.0;
|
|
int joint[]; ArrayResize(joint, MI_BINS * 3); ArrayInitialize(joint, 0);
|
|
int px[]; ArrayResize(px, MI_BINS); ArrayInitialize(px, 0);
|
|
int py[]; ArrayResize(py, 3); ArrayInitialize(py, 0);
|
|
for(int i = 0; i < n; i++)
|
|
{
|
|
//--- rank via binary search on the sorted copy; ties land in the same bin, which is correct
|
|
int lo = 0, hi = n - 1, rank = 0;
|
|
while(lo <= hi)
|
|
{
|
|
int mid = (lo + hi) / 2;
|
|
if(sorted[mid] < vals[i])
|
|
{
|
|
rank = mid + 1;
|
|
lo = mid + 1;
|
|
}
|
|
else
|
|
hi = mid - 1;
|
|
}
|
|
int bx = (int)((double)rank * MI_BINS / n);
|
|
if(bx >= MI_BINS)
|
|
bx = MI_BINS - 1;
|
|
int by = labels[i];
|
|
if(by < 0 || by > 2)
|
|
continue;
|
|
joint[bx * 3 + by]++;
|
|
px[bx]++;
|
|
py[by]++;
|
|
}
|
|
double mi = 0.0;
|
|
for(int b = 0; b < MI_BINS; b++)
|
|
{
|
|
if(px[b] <= 0)
|
|
continue;
|
|
for(int c = 0; c < 3; c++)
|
|
{
|
|
int j = joint[b * 3 + c];
|
|
if(j <= 0 || py[c] <= 0)
|
|
continue;
|
|
double pxy = (double)j / n;
|
|
mi += pxy * MathLog(pxy / (((double)px[b] / n) * ((double)py[c] / n)));
|
|
}
|
|
}
|
|
return (mi > 0.0) ? mi : 0.0;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Scores the CURRENT indicator parameters by how much the resulting |
|
|
//| feature vector tells us about the label - the mean per-column |
|
|
//| mutual information over a stratified sample of in-sample bars. |
|
|
//| |
|
|
//| Deliberately scores EVERY column, not just the ones belonging to |
|
|
//| the parameter being swept. Columns the sweep did not touch |
|
|
//| contribute the SAME amount to every candidate, so they shift the |
|
|
//| mean by a constant and cannot change which candidate wins - while |
|
|
//| avoiding any need for this code to know the feature-vector layout, |
|
|
//| which is exactly the kind of coupling that rots. |
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::BuildMiSample(double &cols[], int &labels[], int labelBarOffset = 0,
|
|
int featureBarOffset = 0, int target = MI_TARGET_BARRIER)
|
|
{
|
|
//--- Continuous targets are collected raw here and discretised after the loop, because equal-frequency
|
|
//--- binning needs the whole sample's distribution before any one row can be assigned a bin.
|
|
double raw[];
|
|
bool continuousTarget = (target != MI_TARGET_BARRIER);
|
|
int bars = m_labelCacheBars;
|
|
if(bars <= 0 || m_neuronsCount <= 0)
|
|
return -1;
|
|
//--- Sample the IS region only. The OOS window must not influence which indicator settings ship, or
|
|
//--- the holdout has been used for selection and stops being a holdout at all.
|
|
int oosCutoff = (int)(MathMax(0, MathMin(100, m_oosSplitPct)) / 100.0
|
|
* MathMax(bars - MathMax(m_historyBars, 0), 0));
|
|
int lo = MathMax(oosCutoff, MathMax(m_barrierHorizonBars, 1) + 1);
|
|
int hi = bars - MathMax(m_historyBars, 0) - 1;
|
|
//--- Keep the OFFSET label lookup inside the same bounds as the features, so a shifted scan measures a
|
|
//--- shift and not an edge effect. Widened symmetrically rather than clamping per bar, which would pile
|
|
//--- several sample rows onto the same clamped label and manufacture association out of nothing.
|
|
//--- THE PAD IS FIXED, NOT |labelBarOffset|. Two builds are only comparable row by row if they enumerate
|
|
//--- the same bars with the same stride, and both `lo` and `stride` below are derived from this range -
|
|
//--- so padding by the requested offset would move every row of the offset build. That is exactly what
|
|
//--- broke the positive control: it paired row k of an unshifted build with row k of a build starting
|
|
//--- `offset` bars later, whose label was then shifted a further `offset`, giving a pair 2*offset apart.
|
|
//--- The measured consequence was a control that reported the MI of labels 48 bars apart while claiming
|
|
//--- 24, failed its 5x gate, and voided every MI figure the EA printed.
|
|
int shiftPad = MiShiftPad();
|
|
if(MathAbs(labelBarOffset) > shiftPad || MathAbs(featureBarOffset) > shiftPad)
|
|
return -1; // caller asked for a shift the pad does not cover
|
|
lo += shiftPad;
|
|
hi -= shiftPad;
|
|
if(hi - lo < MI_MIN_SAMPLES)
|
|
return -1;
|
|
int stride = (int)MathMax(1, (hi - lo) / MI_SAMPLE_BARS);
|
|
//--- Published so the positive control can say how many BARS apart two sample rows are without
|
|
//--- recomputing this arithmetic at the call site, where it would silently drift out of agreement.
|
|
m_miStrideBars = stride;
|
|
int cap = (hi - lo) / stride + 1;
|
|
ArrayResize(cols, cap * m_neuronsCount);
|
|
ArrayResize(labels, cap);
|
|
if(continuousTarget)
|
|
ArrayResize(raw, cap);
|
|
int n = 0;
|
|
for(int i = lo; i < hi && n < cap; i += stride)
|
|
{
|
|
//--- Features come from bar i; the LABEL may be taken from a neighbouring bar (labelBarOffset != 0)
|
|
//--- so the caller can scan for a feature/label misalignment - see the alignment scan in
|
|
//--- ReportFeatureLabelInformation(). Both bars must carry a valid label for the row to count.
|
|
int li = i + labelBarOffset;
|
|
if(i >= ArraySize(m_labelCacheHasValue) || !m_labelCacheHasValue[i])
|
|
continue;
|
|
//--- The geometry scan asks "what WOULD this label be under a different barrier?", which by
|
|
//--- definition is not in the cache. Compute it on the spot instead - the cache belongs to the
|
|
//--- configured geometry and a scan must never write to it.
|
|
if(!m_barrierScanLiveLabels && (li < 0 || li >= ArraySize(m_labelCacheHasValue) || !m_labelCacheHasValue[li]))
|
|
continue;
|
|
if(m_barrierScanLiveLabels && (li < MathMax(m_barrierHorizonBars, 1) || li >= bars))
|
|
continue;
|
|
//--- BufferTempData(), NOT BufferTempDataCompute(). The Compute variant APPENDS the bar's features
|
|
//--- to TempData and never touches m_featureCache - only the caching wrapper writes that array. The
|
|
//--- first version of this function called Compute and then read m_featureCache, which
|
|
//--- ReInitADIndicators had just invalidated, so every column read back constant, FeatureColumnMI
|
|
//--- returned 0 for all of them, and all 17 candidates scored exactly 0.0000 nats. The tuner ran for
|
|
//--- 139 s per chart and always reported "no improvement" - a silent no-op that looked like a
|
|
//--- measurement. Read the values back out of TempData, which is where they actually land.
|
|
//--- FEATURE-side shift, distinct from labelBarOffset and not interchangeable with it. Shifting the
|
|
//--- LABEL changes which trade is being predicted, so at any non-zero offset the features sit INSIDE
|
|
//--- the labelled window and the score is lookahead - which is exactly what the alignment scan
|
|
//--- measures and correctly reports (4.7x more knowable 5 bars into a 128-bar window). Shifting the
|
|
//--- FEATURES instead keeps the label pinned to the entry bar and asks the honest question: does the
|
|
//--- state k bars BEFORE the entry still carry information about that entry's outcome? Positive k is
|
|
//--- strictly older (higher series index), so every row stays causal.
|
|
TempData.Clear();
|
|
if(!BufferTempData(i + featureBarOffset) || TempData.Total() < m_neuronsCount)
|
|
continue;
|
|
for(int f = 0; f < m_neuronsCount; f++)
|
|
cols[n * m_neuronsCount + f] = TempData.At(f);
|
|
if(continuousTarget)
|
|
{
|
|
//--- Excursions come from the cache only. The geometry scan's live-relabel path deliberately
|
|
//--- does not feed them: excursions do not depend on SL/TP at all (see the accumulators in
|
|
//--- TripleBarrierLabel), so re-deriving them per candidate geometry would compute the same
|
|
//--- number repeatedly and invite the impression that it varies with the barrier.
|
|
if(li >= ArraySize(m_excUpCache))
|
|
continue;
|
|
double up = m_excUpCache[li];
|
|
double dn = m_excDownCache[li];
|
|
if(!MathIsValidNumber(up) || !MathIsValidNumber(dn))
|
|
continue;
|
|
//--- A bar that TripleBarrierLabel() could not resolve (no valid ATR or close, typically the
|
|
//--- oldest bars) is still flagged as having a label, but its excursions were cleared to zero
|
|
//--- rather than measured. Price cannot genuinely travel zero in BOTH directions over a whole
|
|
//--- horizon, so this is an unambiguous "not measured" marker. Dropping those rows matters more
|
|
//--- than it looks: under EQUAL-FREQUENCY binning a block of identical zeros drags the lowest
|
|
//--- cut point onto zero, and a third of the sample then lands in one bin carrying no
|
|
//--- information - which would show up as a depressed score and read as "not predictable".
|
|
if(up <= 0.0 && dn <= 0.0)
|
|
continue;
|
|
if(target == MI_TARGET_EXC_UP)
|
|
raw[n] = up;
|
|
else
|
|
if(target == MI_TARGET_EXC_DOWN)
|
|
raw[n] = dn;
|
|
else
|
|
if(target == MI_TARGET_EXC_RANGE)
|
|
raw[n] = up + dn;
|
|
else
|
|
if(target == MI_TARGET_EXC_ASYM)
|
|
raw[n] = up - dn;
|
|
else
|
|
{
|
|
//--- Scale-free asymmetry. The denominator is > 0 here because rows with both
|
|
//--- excursions zero were dropped above, so no guard is needed beyond that.
|
|
raw[n] = (up - dn) / (up + dn); // MI_TARGET_EXC_ASYM_NORM
|
|
}
|
|
labels[n] = 0; // assigned below, once the distribution is known
|
|
}
|
|
else
|
|
if(m_barrierScanLiveLabels)
|
|
{
|
|
ENUM_SIGNAL v = TripleBarrierLabel(li);
|
|
if(v == Neutral && m_lastBarrierTimedOut)
|
|
m_barrierScanTimeouts++;
|
|
labels[n] = (v == Buy) ? 0 : ((v == Sell) ? 1 : 2);
|
|
}
|
|
else
|
|
labels[n] = m_labelCacheBuy[li] ? 0 : (m_labelCacheSell[li] ? 1 : 2);
|
|
n++;
|
|
}
|
|
TempData.Clear();
|
|
//--- EQUAL-FREQUENCY DISCRETISATION into the same 3 classes FeatureColumnMI's joint table expects, so
|
|
//--- every downstream piece - the block permutation, the null, the p-value, the lag profile - works on
|
|
//--- a continuous target with no change at all. Equal-frequency rather than equal-width because these
|
|
//--- distributions are fat-tailed (MFE especially): fixed-width bins would put almost every row in the
|
|
//--- first bin and measure nothing. It also fixes H(Y) at ln(3) = 1.099 nats for all four excursion
|
|
//--- targets, which makes their scores directly comparable to each other AND to the barrier label's
|
|
//--- ~1.02 - a comparison that would otherwise be confounded by class balance.
|
|
if(continuousTarget && n > 0)
|
|
{
|
|
double sorted[];
|
|
ArrayResize(sorted, n);
|
|
ArrayCopy(sorted, raw, 0, 0, n);
|
|
ArraySort(sorted);
|
|
double cut1 = sorted[n / 3];
|
|
double cut2 = sorted[(2 * n) / 3];
|
|
//--- A degenerate target (every value identical, e.g. a cache that never filled) would land every
|
|
//--- row in one class and score a flat zero. Say so rather than reporting the zero as a finding.
|
|
if(cut1 == cut2 && sorted[0] == sorted[n - 1])
|
|
{
|
|
Print(ID + ": MI excursion target " + IntegerToString(target) + " is CONSTANT across all "
|
|
+ IntegerToString(n) + " sampled bars - the excursion cache did not fill. Treating as "
|
|
"unusable rather than reporting its zero score as a measurement.");
|
|
return -1;
|
|
}
|
|
for(int q = 0; q < n; q++)
|
|
labels[q] = (raw[q] <= cut1) ? 0 : ((raw[q] <= cut2) ? 1 : 2);
|
|
}
|
|
return n;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Score an already-extracted sample. Split out from the extraction |
|
|
//| above so the permutation test can reuse ONE sample across every |
|
|
//| draw: feature extraction dominates the cost, and re-running it |
|
|
//| per shuffle is what would have made a few hundred permutations |
|
|
//| unaffordable. The shuffle is in place and destructive, which is |
|
|
//| harmless - composing permutations still yields a uniform |
|
|
//| permutation, so successive draws stay independent - but it does |
|
|
//| mean the OBSERVED (unshuffled) statistic must be taken first. |
|
|
//+------------------------------------------------------------------+
|
|
double CExpertSignalAIBase::ScoreMiSample(const double &cols[], int &labels[], int n, bool shuffleLabels)
|
|
{
|
|
if(n < MI_MIN_SAMPLES)
|
|
return -1.0;
|
|
//--- PERMUTATION BASELINE. Mutual information estimated from finite samples is biased UPWARD - with
|
|
//--- MI_BINS bins and 3 classes the bias is roughly (bins-1)(classes-1)/(2n) nats, which at these
|
|
//--- sample sizes is the same order as any real edge in this domain. So a raw MI figure is
|
|
//--- uninterpretable on its own: 0.004 nats could be a genuine weak signal or could be pure noise.
|
|
//--- Shuffling the labels destroys every real association while leaving the sample size, the binning
|
|
//--- and the class proportions untouched, so the score it produces IS this dataset's noise floor,
|
|
//--- measured rather than approximated. Reporting the two together turns "0.0042 nats" into either
|
|
//--- "0.0042 against a 0.0041 floor" (nothing) or "0.0042 against a 0.0009 floor" (something).
|
|
//--- BLOCK permutation, not a free one, and the difference is the whole validity of the test.
|
|
//--- Triple-barrier labels OVERLAP: two sample rows less than m_barrierHorizonBars apart share most of
|
|
//--- their outcome window, so their labels are strongly dependent. A free Fisher-Yates shuffle destroys
|
|
//--- that dependence as well as the feature/label association, which makes the null distribution far
|
|
//--- NARROWER than the truth and hands out significance that isn't there. The 2026-08-01 symbol sweep
|
|
//--- showed it in the raw: excess tracked the sampling STRIDE almost monotonically, and the three D1
|
|
//--- cells - where the stride had collapsed to 1-5 bars against a 128-bar horizon, i.e. ~99% window
|
|
//--- overlap - returned 5-9x the "signal" of every H1 cell at p=0.005. That was label autocorrelation
|
|
//--- leaking through an independence assumption, not an edge. It is Lopez de Prado ch. 4's non-IID
|
|
//--- problem arriving through the back door of the significance test.
|
|
//--- Permuting whole CONTIGUOUS BLOCKS at least one horizon long preserves the autocorrelation inside a
|
|
//--- block while destroying any feature/label association across blocks - so the null keeps the
|
|
//--- dependence structure and the p-value means what it says. It also degrades honestly: when overlap is
|
|
//--- severe there are few blocks, the null is correspondingly wide, and nothing reaches significance,
|
|
//--- which is the correct answer rather than a flattering one.
|
|
if(shuffleLabels)
|
|
{
|
|
int blockRows = (m_miStrideBars > 0)
|
|
? (int)MathCeil((double)MathMax(m_barrierHorizonBars, 1) / m_miStrideBars) : 1;
|
|
if(blockRows < 1)
|
|
blockRows = 1;
|
|
if(blockRows > n)
|
|
blockRows = n;
|
|
int blocks = (n + blockRows - 1) / blockRows;
|
|
m_miNullBlocks = blocks;
|
|
//--- Fisher-Yates over BLOCK ORDER; within-block order is left untouched, which is what preserves
|
|
//--- the local dependence. Copied out rather than swapped in place because blocks are not
|
|
//--- interchangeable in size - the last one is short whenever blockRows does not divide n.
|
|
int order[];
|
|
ArrayResize(order, blocks);
|
|
for(int b = 0; b < blocks; b++)
|
|
order[b] = b;
|
|
for(int b = blocks - 1; b > 0; b--)
|
|
{
|
|
//--- ShuffleRandomIndex, not MathRand()%: with blockRows == 1 the block count equals the row
|
|
//--- count, which can exceed MathRand()'s 15-bit range - same bias as the pass-2 queue shuffle.
|
|
int j = ShuffleRandomIndex(b + 1);
|
|
int t = order[b];
|
|
order[b] = order[j];
|
|
order[j] = t;
|
|
}
|
|
int shuffled[];
|
|
ArrayResize(shuffled, n);
|
|
int w = 0;
|
|
for(int b = 0; b < blocks && w < n; b++)
|
|
{
|
|
int src = order[b] * blockRows;
|
|
for(int q = 0; q < blockRows && w < n; q++)
|
|
{
|
|
int s = src + q;
|
|
shuffled[w++] = (s < n) ? labels[s] : labels[n - 1];
|
|
}
|
|
}
|
|
for(int i = 0; i < n; i++)
|
|
labels[i] = shuffled[i];
|
|
}
|
|
//--- H(Y) over the sampled labels, so the caller can express MI as a fraction of the information the
|
|
//--- label actually contains. Computed AFTER any shuffle, which leaves it unchanged by construction
|
|
//--- (a permutation preserves the class counts) - that invariance is itself a check on the shuffle.
|
|
int classCount[3] = {0, 0, 0};
|
|
for(int k = 0; k < n; k++)
|
|
classCount[labels[k]]++;
|
|
m_miLabelEntropy = 0.0;
|
|
for(int c = 0; c < 3; c++)
|
|
{
|
|
if(classCount[c] <= 0)
|
|
continue;
|
|
double pc = (double)classCount[c] / n;
|
|
m_miLabelEntropy -= pc * MathLog(pc);
|
|
}
|
|
double colVals[];
|
|
ArrayResize(colVals, n);
|
|
double total = 0.0;
|
|
m_miBestColumn = 0.0;
|
|
for(int f = 0; f < m_neuronsCount; f++)
|
|
{
|
|
for(int k = 0; k < n; k++)
|
|
colVals[k] = cols[k * m_neuronsCount + f];
|
|
double mi = FeatureColumnMI(colVals, labels, n);
|
|
total += mi;
|
|
if(mi > m_miBestColumn)
|
|
m_miBestColumn = mi;
|
|
}
|
|
return total / m_neuronsCount;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Extract + score in one call - the form the coordinate sweep uses, |
|
|
//| where each candidate genuinely needs a fresh extraction because |
|
|
//| the indicator settings (and therefore the features) just changed. |
|
|
//+------------------------------------------------------------------+
|
|
double CExpertSignalAIBase::ScoreCurrentParamsByMI(bool shuffleLabels = false)
|
|
{
|
|
double cols[];
|
|
int labels[];
|
|
int n = BuildMiSample(cols, labels);
|
|
if(n < MI_MIN_SAMPLES)
|
|
return -1.0;
|
|
return ScoreMiSample(cols, labels, n, shuffleLabels);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| FILTER-BASED indicator tuning. Replaced the genetic + successive- |
|
|
//| halving search on 2026-08-01. |
|
|
//| |
|
|
//| WHY THE GA HAD TO GO - measured, not assumed. Its cost was |
|
|
//| population x generations x rungs x seeds x eras-per-rung: |
|
|
//| rung 0: 8 cand x 3 seeds x 3 eras = 72 eras |
|
|
//| rung 1: 4 cand x 3 seeds x 8 eras = 96 |
|
|
//| rung 2: 2 cand x 3 seeds x 20 eras = 120 |
|
|
//| = 288 eras per generation x 4 generations = 1152 eras |
|
|
//| BEFORE the winner's real training started. Measured on SP500 H1: |
|
|
//| 9.3 h for the perceptron, 13.2 h for conv, ~48 h for LSTM and |
|
|
//| hybrid. Two days to tune is not a first-run experience. |
|
|
//| |
|
|
//| And it bought nothing. The space here is 90 points (10 MA periods |
|
|
//| x 9 MA types), so 1152 evaluations revisited each point ~13 times; |
|
|
//| meanwhile rungs of 3 and 8 eras cannot separate two MA periods at |
|
|
//| all - the 2026-08-01 run's finalists all scored 25.0-25.9% |
|
|
//| balanced accuracy, i.e. indistinguishable noise, and it then |
|
|
//| deployed the "winner" of that. |
|
|
//| |
|
|
//| THE REAL ERROR was using a full training run as the scoring |
|
|
//| function for a feature's period. The reference book does not: ch. |
|
|
//| 3.3 selects inputs by measuring each candidate indicator's |
|
|
//| CORRELATION with the target and dropping the ones with none, with |
|
|
//| no network involved. Mutual information is the same idea without |
|
|
//| the linearity assumption, which matters here because the label is |
|
|
//| 3-class categorical and the features are not monotonically related |
|
|
//| to it. Scoring is then arithmetic over cached features: seconds, |
|
|
//| not hours, and it scales with the number of enabled features |
|
|
//| rather than with topology cost - so LSTM tunes as fast as the MLP. |
|
|
//| |
|
|
//| COORDINATE SWEEP, not a product sweep: each parameter is optimised |
|
|
//| against the others' current values, one at a time. Cost is the SUM |
|
|
//| of the per-parameter candidate counts, not their product, so |
|
|
//| enabling every indicator stays affordable. Two passes, because the |
|
|
//| second can exploit what the first learned about the others; it |
|
|
//| stops early the moment a pass changes nothing. |
|
|
//| |
|
|
//| HONEST LIMIT, stated because it is the price of the trade: MI is a |
|
|
//| MARGINAL measure. It scores each feature column on its own, so a |
|
|
//| parameter that only pays off in combination with another can be |
|
|
//| missed. That is the standard filter-vs-wrapper tradeoff (Guyon & |
|
|
//| Elisseeff 2003). Given the wrapper here was ranking pure noise at |
|
|
//| 48 h a run, a fast marginal score is strictly the better deal. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::TuneIndicatorsByFilter(void)
|
|
{
|
|
double best[];
|
|
m_indicatorTuner.Flatten(best);
|
|
double bestScore = ScoreCurrentParamsByMI();
|
|
if(bestScore < 0.0)
|
|
{
|
|
Print(ID + ": auto-tune skipped - not enough labelled in-sample bars to score indicator settings");
|
|
return;
|
|
}
|
|
double startScore = bestScore;
|
|
int evaluated = 0;
|
|
uint t0 = GetTickCount();
|
|
//--- SPREAD OF THE CANDIDATE SCORES. Without it "no improvement" is ambiguous between two readings
|
|
//--- that want opposite responses: INERT (trial scores identical to the incumbent because the
|
|
//--- parameter change never reaches the scored features, so `sc > bestScore` can never fire) versus
|
|
//--- LIVE and genuinely finding nothing. A spread of exactly zero says the first; a spread near the
|
|
//--- estimator's own noise says the second - and then the winner needs the family-wise gate the
|
|
//--- geometry scan and lag profile now carry, because installing a winner CHANGES THE FEATURE VECTOR
|
|
//--- and forces a fresh topology, a far heavier consequence than a printed row.
|
|
//---
|
|
//--- This measures the distinction directly, which is the point: the run-to-run evidence cannot settle
|
|
//--- it. "No improvement" on four consecutive runs (2026-08-05/06, 17 candidates) looks damning if the
|
|
//--- runs are treated as independent trials, but they are NOT - the scorer is deterministic and the
|
|
//--- runs cover nearly the same bars, so an incumbent that is the maximum on this data is the maximum
|
|
//--- on every run. That is one ~1-in-18 observation with three correlated repeats, not four of them.
|
|
//--- Note also that the INERT failure has already happened once here in a different form and was
|
|
//--- fixed (see the BufferTempData note in BuildMiSample: every candidate scored exactly 0.0000).
|
|
//--- Non-zero scores now mean that particular fault is gone.
|
|
double candMin = DBL_MAX, candMax = -DBL_MAX;
|
|
int readyMin = INT_MAX;
|
|
//--- The configured settings, kept so a winner that fails the gate below can be handed back. best[] is
|
|
//--- mutated in place by the descent, so it cannot serve as the restore point.
|
|
double configured[];
|
|
ArrayCopy(configured, best);
|
|
for(int pass = 0; pass < MI_TUNE_PASSES; pass++)
|
|
{
|
|
bool improvedThisPass = false;
|
|
for(int p = 0; p < AD_TUNE_PARAM_COUNT; p++)
|
|
{
|
|
//--- skip parameters whose indicator is switched off - they cannot affect the feature vector
|
|
int owner = m_indicatorTuner.ParamOwner(p);
|
|
bool on = (owner == 0 && m_useADCumulativeDelta) || (owner == 1 && m_useADShorteningOfThrust) ||
|
|
(owner == 2 && m_useADWyckoffEventStream) || (owner == 3 && m_useADWyckoffFailedStructure) ||
|
|
(owner == 4 && m_useADWyckoffSignificantBarInversion) || (owner == 5 && m_useMA) ||
|
|
(owner == 6 && m_useRSI) || (owner == 7 && m_useMACD) || (owner == 8 && m_useIchimoku);
|
|
if(!on)
|
|
continue;
|
|
double cands[];
|
|
int nc = m_indicatorTuner.ParamCandidates(p, cands);
|
|
double keep = best[p];
|
|
for(int c = 0; c < nc; c++)
|
|
{
|
|
if(cands[c] == keep)
|
|
continue; // already scored as the incumbent
|
|
double trial[];
|
|
ArrayCopy(trial, best);
|
|
trial[p] = cands[c];
|
|
m_indicatorTuner.Unflatten(trial);
|
|
ReInitADIndicators(m_indicatorsPtr); // also invalidates the feature cache (params changed)
|
|
//--- REFRESH, or the re-init changes nothing that the scorer can see. ReInitADIndicators
|
|
//--- creates a NEW handle carrying the new parameters and flags the feature cache stale, so
|
|
//--- features are genuinely recomputed - but BufferTempDataCompute() reads the CIndicatorBuffer
|
|
//--- objects, and only Refresh() copies data out of a handle into those. Without this the
|
|
//--- buffers still hold values copied from the PREVIOUS handle, so every candidate is scored on
|
|
//--- identical features. Measured on SP500 H1 2026-08-07: all 17 candidates returned exactly
|
|
//--- 0.00359 nats, a candidate-score span of 0.00000.
|
|
RefreshData();
|
|
int ready = TunableBarsCalculated();
|
|
if(ready >= 0)
|
|
readyMin = (int)MathMin(readyMin, ready);
|
|
double sc = ScoreCurrentParamsByMI();
|
|
evaluated++;
|
|
if(sc >= 0.0)
|
|
{
|
|
candMin = MathMin(candMin, sc);
|
|
candMax = MathMax(candMax, sc);
|
|
}
|
|
if(sc > bestScore)
|
|
{
|
|
bestScore = sc;
|
|
keep = cands[c];
|
|
improvedThisPass = true;
|
|
}
|
|
}
|
|
best[p] = keep;
|
|
}
|
|
if(!improvedThisPass)
|
|
break; // coordinate descent has converged - further passes cannot move anything
|
|
}
|
|
//--- SELECTION GATE. bestScore is a MAXIMUM over every candidate scored, so it carries the same defect
|
|
//--- the barrier-geometry winner test and the lag profile were fixed for: the maximum of N draws from a
|
|
//--- null sits well above any single draw, and installing on "it beat the incumbent" alone crowns noise.
|
|
//--- The stakes here are higher than either of those, because this one ACTS - it replaces the user's
|
|
//--- deliberate indicator settings and forces BuildFreshTopology(), so the network then trains on
|
|
//--- whatever the noise picked.
|
|
//---
|
|
//--- Test: draw the winner's own permutation null once (the sample is extracted once and every draw
|
|
//--- reshuffles it - see ScoreMiSample), take the per-candidate p, then correct it for having CHOSEN
|
|
//--- this candidate out of N with Sidak: p_family = 1 - (1 - p)^N. Sidak rather than an explicit
|
|
//--- max-of-N resample because each candidate here has a DIFFERENT feature set, so their draws cannot
|
|
//--- be pooled the way the geometry scan's can; Sidak needs only the one null and is exact under
|
|
//--- independence, mildly anti-conservative under positive dependence - stated rather than hidden.
|
|
//---
|
|
//--- WHAT THIS DOES NOT ESTABLISH: that the winner beats the INCUMBENT by a significant margin. It
|
|
//--- bounds the "best of N noise draws" failure, which is the one that was actually live here. Requiring
|
|
//--- bestScore > startScore as well means a change needs both an improvement and a defensible signal.
|
|
bool install = (bestScore > startScore);
|
|
double pFamily = 1.0;
|
|
int distinct = (int)MathMax(evaluated + 1, 1); // candidates scored, plus the incumbent
|
|
if(install)
|
|
{
|
|
double wc[];
|
|
int wl[];
|
|
int wn = BuildMiSample(wc, wl);
|
|
if(wn >= MI_MIN_SAMPLES)
|
|
{
|
|
double obs = ScoreMiSample(wc, wl, wn, false);
|
|
int atLeast = 0, draws = 0;
|
|
for(int s = 0; s < MI_NOISE_PERMUTATIONS; s++)
|
|
{
|
|
double d = ScoreMiSample(wc, wl, wn, true);
|
|
if(d < 0.0)
|
|
continue;
|
|
if(d >= obs)
|
|
atLeast++;
|
|
draws++;
|
|
}
|
|
if(draws > 0)
|
|
{
|
|
double pSingle = (double)(1 + atLeast) / (draws + 1);
|
|
pFamily = 1.0 - MathPow(1.0 - pSingle, (double)distinct);
|
|
}
|
|
}
|
|
install = (pFamily <= MI_TUNE_ALPHA);
|
|
}
|
|
if(!install)
|
|
{
|
|
ArrayCopy(best, configured);
|
|
bestScore = startScore;
|
|
}
|
|
//--- install the winner and leave the indicators/feature cache consistent with it
|
|
m_indicatorTuner.Unflatten(best);
|
|
ReInitADIndicators(m_indicatorsPtr);
|
|
RefreshData();
|
|
double candSpread = (evaluated > 0 && candMax >= candMin) ? (candMax - candMin) : 0.0;
|
|
Print(ID + StringFormat(": auto-tune complete - %d candidate settings scored in %.1fs, "
|
|
"feature/label mutual information %.5f -> %.5f nats%s | candidate scores span "
|
|
"%.5f (%.5f..%.5f)%s",
|
|
evaluated, (GetTickCount() - t0) / 1000.0, startScore, bestScore,
|
|
(bestScore <= startScore ? " (no improvement - keeping the configured settings)" : ""),
|
|
candSpread, (evaluated > 0 ? candMin : 0.0), (evaluated > 0 ? candMax : 0.0),
|
|
(evaluated > 0 && candSpread <= 0.0
|
|
? StringFormat(" <-- ZERO SPREAD: every candidate scored identically, so the "
|
|
"parameter change is STILL not reaching the scored features even "
|
|
"with the post-re-init RefreshData(). Least-ready tunable handle "
|
|
"had %d bars calculated - if that is 0 or far below the study "
|
|
"window, the handles are simply not done calculating yet and the "
|
|
"tuner needs to yield between candidates rather than score them "
|
|
"back to back.", (readyMin == INT_MAX ? -1 : readyMin))
|
|
: StringFormat(" | winner %s (selection p=%.4f after correcting for %d "
|
|
"candidates, need <=%.2f)",
|
|
(install ? "INSTALLED" : "REJECTED - keeping the configured "
|
|
"settings, since the best of N noise draws beats its incumbent "
|
|
"almost every time"),
|
|
pFamily, distinct, MI_TUNE_ALPHA))));
|
|
//--- An EXACTLY zero score is not a weak feature set, it is a broken measurement. Mutual information
|
|
//--- estimated from finite samples is biased UPWARD - roughly (bins-1)(classes-1)/(2N) nats, ~0.0035
|
|
//--- here - so even columns of pure noise score above zero. Landing on 0.0000 means every column read
|
|
//--- back constant, which is what a feature-extraction fault looks like. Said out loud because the
|
|
//--- first version of this function did exactly that and reported it as "no improvement".
|
|
if(bestScore <= 0.0)
|
|
Print(ID + ": WARNING - every candidate scored 0.0000 nats. Finite-sample bias alone should put "
|
|
"noise above zero, so this indicates the feature values are not being read, not that the "
|
|
"features are uninformative. Indicator settings left at their configured values.");
|
|
ReportFeatureLabelInformation();
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| "Do these features predict this label at all?" - answered without |
|
|
//| training, topology or convergence, so unlike every accuracy |
|
|
//| number in this codebase it cannot be confounded by an optimizer |
|
|
//| or an objective. |
|
|
//| |
|
|
//| DELIBERATELY SEPARATE FROM THE TUNER, and not gated on era 0 with |
|
|
//| it. The sweep must only run on a fresh model - re-tuning would |
|
|
//| change the input vector out from under weights already fitted to |
|
|
//| the old one - but this reads the same cached features and writes |
|
|
//| nothing, so tying it to that gate meant the only way to see the |
|
|
//| answer was to bin a model mid-run (45 trained eras, on 2026-08-01) |
|
|
//| purely to re-ask a read-only question. Runs once per attach. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ReportFeatureLabelInformation(void)
|
|
{
|
|
m_miReportDone = true;
|
|
//--- PERMUTATION TEST, done properly. Build the current settings' sample ONCE, take the observed
|
|
//--- statistics from it, then reuse that same sample for every null draw - extraction is the expensive
|
|
//--- part, so this makes a few hundred permutations cost about what five used to.
|
|
//---
|
|
//--- Five was not enough, and the 2026-08-01 log is the proof: all four charts scored the IDENTICAL
|
|
//--- 0.00401 nats on identical features and identical labels, yet reported z of +1.3, +2.0, +4.0 and
|
|
//--- +4.7 - two "at the noise floor", two "real". The whole swing came from estimating the null's spread
|
|
//--- from five draws, where the standard deviation of the standard-deviation estimate is ~35%. The
|
|
//--- denominator was noisier than the effect.
|
|
//---
|
|
//--- So: no z-score and no normality assumption. An EMPIRICAL p-value, counting how many null draws
|
|
//--- reached the observed value, with the +1/(B+1) correction (Phipson & Smyth 2010) that keeps p from
|
|
//--- ever being reported as exactly zero - the test can only ever bound p below by 1/(B+1).
|
|
double cols[];
|
|
int labels[];
|
|
int nSample = BuildMiSample(cols, labels);
|
|
double observed = (nSample >= MI_MIN_SAMPLES) ? ScoreMiSample(cols, labels, nSample, false) : -1.0;
|
|
double signalBestCol = m_miBestColumn;
|
|
double labelEntropy = m_miLabelEntropy;
|
|
double floorSum = 0.0, floorSumSq = 0.0, floorBestColSum = 0.0;
|
|
int draws = 0, atLeastMean = 0, atLeastBestCol = 0;
|
|
uint tPerm = GetTickCount();
|
|
for(int s = 0; observed >= 0.0 && s < MI_NOISE_PERMUTATIONS; s++)
|
|
{
|
|
double sc = ScoreMiSample(cols, labels, nSample, true);
|
|
if(sc < 0.0)
|
|
continue;
|
|
floorSum += sc;
|
|
floorSumSq += sc * sc;
|
|
floorBestColSum += m_miBestColumn;
|
|
if(sc >= observed)
|
|
atLeastMean++;
|
|
//--- The MAX over columns is compared against the null distribution OF THE MAX, which corrects for
|
|
//--- testing 26 features at once by construction - no Bonferroni needed, and far less conservative.
|
|
if(m_miBestColumn >= signalBestCol)
|
|
atLeastBestCol++;
|
|
draws++;
|
|
}
|
|
double floorMean = (draws > 0) ? floorSum / draws : -1.0;
|
|
double floorVar = (draws > 1) ? MathMax(0.0, floorSumSq / draws - floorMean * floorMean) : 0.0;
|
|
double floorSd = MathSqrt(floorVar * (draws > 1 ? (double)draws / (draws - 1) : 1.0));
|
|
double floorBestCol = (draws > 0) ? floorBestColSum / draws : -1.0;
|
|
double pMean = (draws > 0) ? (double)(1 + atLeastMean) / (draws + 1) : 1.0;
|
|
double pBestCol = (draws > 0) ? (double)(1 + atLeastBestCol) / (draws + 1) : 1.0;
|
|
//--- Two SEPARATE questions, because at these sample sizes a small p can accompany a worthless effect.
|
|
//--- (1) Is it real - the p-values. (2) Is it big enough to trade - the excess as a share of H(Y), i.e.
|
|
//--- of everything there is to know about the label. Both are printed; neither is collapsed into a verdict
|
|
//--- that hides the other.
|
|
double excessShare = (labelEntropy > 1e-9 && floorMean >= 0.0)
|
|
? 100.0 * (observed - floorMean) / labelEntropy : 0.0;
|
|
string verdict = (draws > 0 && pMean <= 0.05)
|
|
? "above the noise floor - a real association"
|
|
: "AT THE NOISE FLOOR - indistinguishable from shuffled labels";
|
|
//--- Name the feature vector this was measured on. These numbers are only about the model if the two
|
|
//--- match, and on 2026-08-02 they did not: the report ran before the cross-asset panel existed and
|
|
//--- silently described a narrower vector than training used. Stating the width and the panel's
|
|
//--- presence makes that mismatch visible in the log instead of requiring a timestamp comparison.
|
|
string vecNote = StringFormat("%d features/bar, cross-asset %s", m_neuronsCount,
|
|
m_crossAsset.IsReady()
|
|
? "PRESENT"
|
|
: "ABSENT (reference symbols unsynchronised - these numbers describe "
|
|
"a NARROWER vector than training will use)");
|
|
Print(ID + StringFormat(": feature/label information - %.5f nats/feature vs a shuffled-label null of "
|
|
"%.5f +/- %.5f over %d permutations, p=%.4f; strongest single feature %.5f vs "
|
|
"%.5f (null max, p=%.4f); excess is %.2f%% of the label's %.3f nats of entropy "
|
|
"(%d samples %d bars apart = %d independent blocks over a %d-bar horizon, "
|
|
"%.1fs) [%s]. %s.",
|
|
observed, floorMean, floorSd, draws, pMean,
|
|
signalBestCol, floorBestCol, pBestCol, excessShare, labelEntropy,
|
|
nSample, m_miStrideBars, m_miNullBlocks, m_barrierHorizonBars,
|
|
(GetTickCount() - tPerm) / 1000.0, vecNote, verdict));
|
|
//--- POWER, stated up front. The block permutation above makes the p-value HONEST under overlapping
|
|
//--- labels, but it cannot manufacture information that overlap destroyed: when the sampling stride is
|
|
//--- far shorter than the horizon there are few genuinely independent blocks, and a handful of blocks
|
|
//--- cannot resolve an effect this small however many rows they contain. Saying so prevents the opposite
|
|
//--- error to the one this replaced - reading "not significant" as "no signal" when it means "not enough
|
|
//--- independent data to tell".
|
|
if(m_miNullBlocks > 0 && m_miNullBlocks < 30)
|
|
Print(ID + StringFormat(": NOTE - only %d independent label blocks in this sample (%d-bar horizon, "
|
|
"%d-bar sampling stride). The rows overlap heavily, so this test has little "
|
|
"power: treat a non-significant result here as 'not enough independent "
|
|
"history to answer', not as 'no signal'. More history, or a shorter horizon, "
|
|
"is what would settle it.", m_miNullBlocks, m_barrierHorizonBars,
|
|
m_miStrideBars));
|
|
//--- Stated every time, not only on a bad result: this measure is MARGINAL and PER-BAR, while the network
|
|
//--- reads m_historyBars bars at once. It can therefore only ever prove that signal EXISTS, never that it
|
|
//--- does not - an interaction across features or across time is invisible to it by construction. Said
|
|
//--- out loud so a floor-level reading is not over-read into "this instrument is unpredictable".
|
|
if(!(draws > 0 && pMean <= 0.05))
|
|
Print(ID + ": NOTE - that measure is marginal (one feature at a time) and per-bar, whereas the "
|
|
"network sees " + IntegerToString((int)m_historyBars) + " bars jointly. A floor-level reading "
|
|
"rules out a simple per-feature edge; it cannot rule out one that only exists in combination "
|
|
"or across time. It does mean no per-feature indicator retuning will help.");
|
|
if(observed < 0.0)
|
|
return;
|
|
//--- POSITIVE CONTROL. Three separate "measurements" in this codebase have turned out to be silent
|
|
//--- no-ops that produced plausible numbers (the MI scorer reading an array nobody filled; the
|
|
//--- eval-mode guard that switched off the imbalance correction; the alternation gate whose premise was
|
|
//--- never true). A floor reading is therefore worthless until the instrument is shown to respond to a
|
|
//--- signal that is KNOWN to be there. This one is free: the label of a NEIGHBOURING sample row. Rows are
|
|
//--- `stride` bars apart, far inside the barrier horizon, so their outcome windows overlap heavily and
|
|
//--- the two labels must be strongly associated. Fed through the identical binning and estimator as every
|
|
//--- other column. If THIS lands near the floor, the estimator is broken and no MI number above means
|
|
//--- anything; if it lands far above, a floor reading on the real features can be believed.
|
|
//--- The control pairs each row's label with the label of a bar a FIXED, KNOWN distance away, so the two
|
|
//--- outcome windows overlap heavily and must be strongly associated.
|
|
//--- THIS CONTROL HAS NOW CRIED WOLF TWICE, AND BOTH TIMES THE ESTIMATOR WAS INNOCENT.
|
|
//--- 2026-08-01 it paired with the NEXT SAMPLE ROW, whose distance is the sampling stride - and stride
|
|
//--- varies with how much history a symbol has, so the control's strength varied with the
|
|
//--- cell rather than with the estimator. All three M5 cells (stride 160-717 bars against a
|
|
//--- 128-bar horizon, i.e. windows that do not overlap AT ALL) voided their own results.
|
|
//--- 2026-08-02 the range was padded by |offset|, which moved the offset build's FIRST BAR as well as
|
|
//--- its label, so row k of one build sat `offset` bars from row k of the other and the
|
|
//--- label was shifted a further `offset`: the pair was 2x as far apart as reported. On
|
|
//--- SP500 H1 it printed 0.00307 nats for "24 bars apart" - which is the true value for 48
|
|
//--- bars - failed its 5x gate, and stamped "every mutual-information figure above is void"
|
|
//--- on measurements that were fine. Confirmed by computing the same quantity independently
|
|
//--- in research/test_mi_control.py: 0.01655 at 24 bars, 0.00298 at 48.
|
|
//--- The lesson both share: A CONTROL THAT DEPENDS ON THE THING IT CERTIFIES CANNOT CERTIFY IT. Pin the
|
|
//--- control's distance to something the data cannot move, and make it a distance where the association
|
|
//--- is overwhelming rather than marginal - hence the adjacent bar below.
|
|
//--- TWO distances, and the GATE is the adjacent bar. Its barrier window overlaps the reference one by
|
|
//--- (h-1)/h, so "these must be associated" is unarguable, and unlike a horizon-relative offset it does
|
|
//--- not vary with the horizon, the stride or the symbol. The quarter-horizon figure is kept as a
|
|
//--- DIAGNOSTIC because it says something the gate cannot: how fast a triple-barrier label decorrelates.
|
|
//--- Measured independently on SP500 H1 (research/test_mi_control.py, 74k bars): 0.542 nats at 1 bar,
|
|
//--- 0.017 at 24, 0.003 at 48, against a ~0.002 floor. Note what that means - a quarter-horizon control
|
|
//--- clears a 5x gate by under 2x even when everything is working, which is far too little headroom for
|
|
//--- the one measurement whose job is to certify all the others.
|
|
double controlMi = -1.0, decorrMi = -1.0;
|
|
int controlBars = 1;
|
|
int decorrBars = MathMax(1, MathMax(m_barrierHorizonBars, 1) / 4);
|
|
{
|
|
//--- Rebuilt rather than reused because the permutation loop above destroyed the honest label
|
|
//--- ordering, and controlling against a shuffled array would measure the floor twice.
|
|
double c0[], cK[];
|
|
int l0[], lK[];
|
|
int n0 = BuildMiSample(c0, l0);
|
|
if(n0 >= MI_MIN_SAMPLES)
|
|
{
|
|
int offs[2];
|
|
offs[0] = controlBars;
|
|
offs[1] = decorrBars;
|
|
for(int oi = 0; oi < 2; oi++)
|
|
{
|
|
int nK = BuildMiSample(cK, lK, offs[oi]);
|
|
//--- Both builds are padded by the SAME fixed amount, so they enumerate the same bars with
|
|
//--- the same stride and row k of one is row k of the other. Sized from what actually came
|
|
//--- back, never from the caller's count.
|
|
int nc = MathMin(n0, nK);
|
|
if(nc < MI_MIN_SAMPLES)
|
|
continue;
|
|
double neighbourLabel[];
|
|
int selfLabels[];
|
|
ArrayResize(neighbourLabel, nc);
|
|
ArrayResize(selfLabels, nc);
|
|
for(int k = 0; k < nc; k++)
|
|
{
|
|
selfLabels[k] = l0[k];
|
|
neighbourLabel[k] = (double)lK[k];
|
|
}
|
|
double v = FeatureColumnMI(neighbourLabel, selfLabels, nc);
|
|
if(oi == 0)
|
|
controlMi = v;
|
|
else
|
|
decorrMi = v;
|
|
}
|
|
}
|
|
}
|
|
Print(ID + StringFormat(": MI positive control - the ADJACENT bar's label (windows overlap %d of %d bars) "
|
|
"scores %.5f nats against the ~%.5f noise floor; by a quarter horizon (%d bars) "
|
|
"it is already down to %.5f, which is how fast this target decorrelates. %s",
|
|
MathMax(m_barrierHorizonBars, 1) - 1, MathMax(m_barrierHorizonBars, 1),
|
|
controlMi, floorMean, decorrBars, decorrMi,
|
|
(controlMi > floorMean * 5.0)
|
|
? "The estimator detects a known association on this exact data, so a "
|
|
"floor-level reading above is a real finding and not a broken measurement."
|
|
: "WARNING - the estimator FAILED to detect an association that must be there. "
|
|
"Every mutual-information figure above is void; fix this before drawing any "
|
|
"conclusion from them."));
|
|
//--- ALIGNMENT SCAN. A floor reading has two very different causes: the features genuinely do not predict
|
|
//--- this target, or they DO and something upstream has knocked the two out of step (an off-by-one in the
|
|
//--- label index, a horizon applied to the wrong bar, a feature window that lags what it claims). Both
|
|
//--- destroy the information before any topology sees it, and both look identical in every accuracy number
|
|
//--- this EA prints - which is exactly why four different architectures all landed on the same precision.
|
|
//--- Re-scoring against the label taken from bar i+k separates them: a peak at some k != 0 IS a
|
|
//--- misalignment (and names its size), a flat profile says the features simply do not carry this target.
|
|
//--- THE TWO DIRECTIONS ARE NOT SYMMETRIC, and the first version of this scan treated them as if they
|
|
//--- were - it read the k>0 rise as a misalignment and cried "fix this before concluding anything",
|
|
//--- which was a false alarm produced by the diagnostic's own design.
|
|
//---
|
|
//--- Bar indices here are MQL5 SERIES indices: HIGHER index = OLDER bar (TripleBarrierLabel walks its
|
|
//--- window with `for(t = idx-1; t >= idx-horizon; t--)`, i.e. decreasing index = forward in time).
|
|
//--- So:
|
|
//--- k < 0 the label belongs to a NEWER bar, whose barrier window opens AFTER the features exist.
|
|
//--- Nothing at bar i can legitimately know it. A peak here is real LOOKAHEAD and is a bug.
|
|
//--- k > 0 the label belongs to an OLDER bar, whose window is already k bars into its life by the
|
|
//--- time bar i happens - so the features at bar i legitimately contain the realised first k
|
|
//--- bars of that outcome. MI MUST rise with k. That is arithmetic, not a defect.
|
|
//--- Only the k<0 side can indict the pipeline. The k>0 side is a second positive control, and its
|
|
//--- GRADIENT is the useful number: it says how fast a barrier outcome becomes knowable once the window
|
|
//--- is running, against how little is knowable at entry (k=0).
|
|
int offsets[] = { -5, -3, -2, -1, 0, 1, 2, 3, 5 };
|
|
string profile = "";
|
|
double atZero = -1.0, worstFuture = -1.0, farPast = -1.0;
|
|
int worstFutureK = 0;
|
|
for(int oi = 0; oi < ArraySize(offsets); oi++)
|
|
{
|
|
double oc[];
|
|
int ol[];
|
|
int on = BuildMiSample(oc, ol, offsets[oi]);
|
|
double os = (on >= MI_MIN_SAMPLES) ? ScoreMiSample(oc, ol, on, false) : -1.0;
|
|
profile += StringFormat("%s%+d:%.5f", (oi > 0 ? " " : ""), offsets[oi], os);
|
|
if(offsets[oi] == 0)
|
|
atZero = os;
|
|
else
|
|
if(offsets[oi] < 0 && os > worstFuture)
|
|
{
|
|
worstFuture = os;
|
|
worstFutureK = offsets[oi];
|
|
}
|
|
else
|
|
if(offsets[oi] > 0)
|
|
farPast = os; // offsets ascend, so this ends on the largest k
|
|
}
|
|
//--- A MARGIN, not a bare comparison. Every one of these offsets is an estimate with the same noise as
|
|
//--- the headline statistic, so "k=-3 came out above k=0" is meaningless when the gap is smaller than the
|
|
//--- null's own spread. Shipped without this, the 2026-08-01 sweep flagged LOOKAHEAD on 7 of 12 cells on
|
|
//--- gaps of 0.00008-0.00040 nats against a measured null sd of ~0.00030 - all noise, every one. Three
|
|
//--- SDs is the same discipline the deploy floor already applies to precision: an anomaly has to clear
|
|
//--- the measurement error before it gets a name. (Third time this session that comparing two point
|
|
//--- estimates without their spread produced a confident wrong answer - see MI_NOISE_PERMUTATIONS.)
|
|
double lookaheadMargin = 3.0 * floorSd;
|
|
string alignVerdict;
|
|
if(worstFuture > atZero + lookaheadMargin)
|
|
alignVerdict = StringFormat(" | LOOKAHEAD - k=%d (a label whose barrier window opens AFTER these "
|
|
"features exist) scores %.5f against %.5f at k=0, clearing the %.5f "
|
|
"margin (3 sd of the null). The features can only score there by "
|
|
"containing future information. Fix that before trusting any accuracy "
|
|
"number this EA prints.", worstFutureK, worstFuture, atZero, lookaheadMargin);
|
|
else
|
|
alignVerdict = StringFormat(" | clean: no future label (k<0) beats k=0, so there is no lookahead. "
|
|
"The rise on the k>0 side is expected - those windows are already open, "
|
|
"so the features hold part of the answer - and its size is the finding: "
|
|
"%.5f at k=+5 against %.5f at k=0, i.e. ~%.1fx more is knowable %d bars "
|
|
"into a %d-bar window than at the entry the model actually trades.",
|
|
farPast, atZero, (atZero > 1e-9 ? farPast / atZero : 0.0), 5,
|
|
m_barrierHorizonBars);
|
|
Print(ID + ": MI label-alignment scan (label from bar i+k; higher index = OLDER bar, so k<0 is the "
|
|
"future) - " + profile + alignVerdict);
|
|
ReportFeatureLagProfile();
|
|
//--- Runs after the lag profile and before the geometry scan on purpose: the geometry scan chooses
|
|
//--- among SL/TP pairings, and this asks whether predicting SL/TP is a well-posed problem at all.
|
|
//--- Reading them in that order stops a geometry winner from being interpreted as evidence that the
|
|
//--- exit is learnable.
|
|
ReportExcursionInformation();
|
|
ReportBarrierGeometryScan();
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| WHICH BARRIER GEOMETRY IS ACTUALLY PREDICTABLE AT ENTRY. |
|
|
//| |
|
|
//| The alignment scan established the shape of the problem: 4.7x more |
|
|
//| is knowable 5 bars into a 128-bar window than at the entry the |
|
|
//| model trades on. A 6xATR target reached over 128 bars is decided |
|
|
//| overwhelmingly by what happens DURING the window, so whatever the |
|
|
//| entry state knows is buried under 128 bars of subsequent noise. |
|
|
//| That is a property of the TARGET, and no topology can undo it - |
|
|
//| which is why four different architectures all landed on precision |
|
|
//| exactly equal to the base rate. |
|
|
//| |
|
|
//| So measure the target instead of guessing at it. For each SL/TP |
|
|
//| pairing the user can actually select, relabel the same sampled |
|
|
//| bars and score how much the SAME features say about THAT outcome. |
|
|
//| Seconds, no training, no topology. |
|
|
//| |
|
|
//| RANKED ON EXCESS OVER ITS OWN NULL, IN NATS. The first version |
|
|
//| divided that by the geometry's own H(Y), reasoning that each label |
|
|
//| has a different amount of information available to find. That was |
|
|
//| backwards and it produced a wrong answer on the first run: it |
|
|
//| named 3:10, whose horizon is CLAMPED (it wants ~320 bars and gets |
|
|
//| BARRIER_HORIZON_MAX), so most trades never resolve, Neutral |
|
|
//| dominates, H(Y) collapses - and dividing by a collapsing |
|
|
//| denominator made the most degenerate label look like the most |
|
|
//| predictable one. Subtracting each geometry's own measured null |
|
|
//| already removes the class-balance bias, which is the only thing |
|
|
//| the normalisation was needed for. |
|
|
//| |
|
|
//| A clamped geometry is DISQUALIFIED outright, not merely ranked |
|
|
//| down. The deployed EA holds until SL or TP with no bar limit, so a |
|
|
//| truncated label trains the model on a question the strategy never |
|
|
//| asks. Directional share is printed for the same reason: a label |
|
|
//| nobody can trade is not a candidate however well it scores. |
|
|
//| |
|
|
//| What it cannot tell you: chance precision equals the break-even |
|
|
//| win rate at every geometry (both are m/(m+k) under a driftless |
|
|
//| walk), so a tighter target does NOT buy expectancy on its own. It |
|
|
//| buys PREDICTABILITY - a shorter window has less noise piled on top |
|
|
//| of what the entry state knows. The ranking finds where the signal |
|
|
//| is largest; it is still on the model to convert it. |
|
|
//+------------------------------------------------------------------+
|
|
//+------------------------------------------------------------------+
|
|
//| HOW FAR BACK THE FEATURES STILL SAY ANYTHING - see the declaration.|
|
|
//| |
|
|
//| Returns the deepest lag whose score clears the null, or 0 when |
|
|
//| none does. Read-only; the caller decides what to do with it. |
|
|
//| |
|
|
//| The null is redrawn PER LAG rather than measured once and reused. |
|
|
//| Finite-sample MI bias depends on the realised class counts and the |
|
|
//| bin occupancy, and both move with the lag because different rows |
|
|
//| survive the validity checks - so a single shared floor would be |
|
|
//| the right number for lag 0 and the wrong one everywhere else. |
|
|
//| Cost is the reason it is a REDUCED draw count: a full |
|
|
//| MI_NOISE_PERMUTATIONS sweep at every lag is 200 x historyBars |
|
|
//| scorings. The gate below is deliberately crude for the same |
|
|
//| reason - this profile decides a LOOKBACK, not a trade. |
|
|
//+------------------------------------------------------------------+
|
|
//+------------------------------------------------------------------+
|
|
//| IS "OPTIMAL SL/TP" LEARNABLE? Scores the same features against |
|
|
//| four excursion targets instead of the barrier class. |
|
|
//| |
|
|
//| The question this exists to settle: predicting an optimal stop and |
|
|
//| target decomposes into HOW FAR price travels and WHICH WAY it goes |
|
|
//| first, and those two behave nothing alike. Excursion SIZE is a |
|
|
//| volatility question, and volatility clustering is one of the most |
|
|
//| robust regularities in markets - RANGE is included precisely as a |
|
|
//| positive control that SHOULD clear, and a run where it does not is |
|
|
//| evidence the measurement is broken rather than that the market is |
|
|
//| unpredictable. ASYMMETRY is direction wearing different clothes, |
|
|
//| and it is the only one of the four that can produce expectancy. |
|
|
//| |
|
|
//| So the informative outcome is the CONTRAST, not any single number: |
|
|
//| RANGE/UP/DOWN clearing while ASYM sits at the floor says size is |
|
|
//| predictable and order is not - i.e. the payoff of a predicted |
|
|
//| SL/TP is position sizing and drawdown control, not edge. That is |
|
|
//| worth having under prop-firm limits, and it is not a signal. |
|
|
//| Exit management on RANDOM entries already moved the payoff ratio |
|
|
//| 0.92 -> 5.72 with expectancy FLAT, so this would agree with a test |
|
|
//| that has already been run a different way. |
|
|
//| |
|
|
//| Why this is not answered by the existing verdicts: every MI figure |
|
|
//| this project has produced scored the TRIPLE-BARRIER label, which |
|
|
//| is one specific question ("does the target come before the stop at |
|
|
//| this fixed geometry"). A noise-floor result there says nothing |
|
|
//| about whether excursion MAGNITUDE is learnable - different target, |
|
|
//| different answer, and worth measuring before rebuilding a head. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ReportExcursionInformation(void)
|
|
{
|
|
int targets[] = { MI_TARGET_EXC_RANGE, MI_TARGET_EXC_UP, MI_TARGET_EXC_DOWN, MI_TARGET_EXC_ASYM,
|
|
MI_TARGET_EXC_ASYM_NORM };
|
|
string names[] = { "RANGE up+dn (volatility control)", "UP (MFE)", "DOWN (MAE)",
|
|
"ASYMMETRY up-dn (RAW - confounded by volatility, read the NORM line instead)",
|
|
"ASYMMETRY NORMALISED (up-dn)/(up+dn) (THE ONE THAT MATTERS)" };
|
|
bool asymCleared = false, sizeCleared = false, rawAsymCleared = false;
|
|
for(int k = 0; k < ArraySize(targets); k++)
|
|
{
|
|
double cols[];
|
|
int labels[];
|
|
int n = BuildMiSample(cols, labels, 0, 0, targets[k]);
|
|
if(n < MI_MIN_SAMPLES)
|
|
{
|
|
Print(ID + ": MI excursion - " + names[k] + ": not enough usable bars to score");
|
|
continue;
|
|
}
|
|
double observed = ScoreMiSample(cols, labels, n, false);
|
|
if(observed < 0.0)
|
|
continue;
|
|
double floorSum = 0.0;
|
|
int draws = 0, atLeast = 0;
|
|
for(int s = 0; s < MI_NOISE_PERMUTATIONS; s++)
|
|
{
|
|
double sc = ScoreMiSample(cols, labels, n, true);
|
|
if(sc < 0.0)
|
|
continue;
|
|
floorSum += sc;
|
|
if(sc >= observed)
|
|
atLeast++;
|
|
draws++;
|
|
}
|
|
if(draws <= 0)
|
|
continue;
|
|
double floorMean = floorSum / draws;
|
|
double p = (double)(1 + atLeast) / (draws + 1);
|
|
bool clears = (p <= MI_LAG_ALPHA);
|
|
//--- H(Y) is ln(3) by construction (equal-frequency bins), so excess-as-a-share-of-entropy is
|
|
//--- comparable across all four targets and against the barrier label's own figure.
|
|
Print(ID + StringFormat(": MI excursion - %s: %.5f nats/feature vs a block-permuted null of %.5f, "
|
|
"p=%.4f over %d draws%s | %.2f%% of the target's %.3f nats (%d samples)",
|
|
names[k], observed, floorMean, p, draws, (clears ? " <-- CLEARS" : ""),
|
|
100.0 * (observed - floorMean) / MathLog(3.0), MathLog(3.0), n));
|
|
if(targets[k] == MI_TARGET_EXC_ASYM_NORM)
|
|
asymCleared = clears; // the ONLY one a directional claim may rest on
|
|
else
|
|
if(targets[k] == MI_TARGET_EXC_ASYM)
|
|
rawAsymCleared = clears;
|
|
else
|
|
if(clears)
|
|
sizeCleared = true;
|
|
}
|
|
//--- The verdict is the CONTRAST. Spelled out rather than left to be read off five numbers, because
|
|
//--- the wrong reading of "UP clears" is "we can predict profitable trades", and that is precisely
|
|
//--- the inference this report exists to prevent.
|
|
//---
|
|
//--- ORDER MATTERS, and the first version had it wrong: the generic size-not-direction branch was
|
|
//--- tested first, and it is true whenever size clears - i.e. always - so the CONFOUND branch was
|
|
//--- unreachable. Measured 2026-08-07 across three symbols: raw asymmetry cleared on all three while
|
|
//--- normalised collapsed on all three, and the one message that explains why never printed.
|
|
if(asymCleared)
|
|
Print(ID + ": MI excursion VERDICT - NORMALISED ASYMMETRY CLEARS. Scale-free directional "
|
|
"information survives dividing the volatility out, which no barrier-label test has ever "
|
|
"found and which the raw asymmetry could not have established on its own. Before acting: "
|
|
"replicate on instruments NOT used to find it, and check the effect is not concentrated in "
|
|
"one volatility regime. If it holds, this is the first real signal here.");
|
|
else
|
|
if(rawAsymCleared)
|
|
Print(ID + ": MI excursion VERDICT - raw asymmetry cleared but the NORMALISED one did not. That "
|
|
"is the signature of the VOLATILITY CONFOUND, not of direction: up-dn scales with sigma, "
|
|
"so a predictable sigma pushes the value into both outer bins and scores while carrying no "
|
|
"directional content at all - and it does so on every instrument, so replication does not "
|
|
"argue against it. Read the raw line as a restatement of RANGE. Excursion SIZE is "
|
|
"predictable and worth using for position sizing and drawdown control; DIRECTION is not, "
|
|
"so no SL/TP head can create expectancy. Agrees with the random-entry exit test (payoff "
|
|
"ratio 0.92->5.72, expectancy flat).");
|
|
else
|
|
if(sizeCleared)
|
|
Print(ID + ": MI excursion VERDICT - excursion SIZE is predictable, DIRECTION is not. A model "
|
|
"trained to output SL/TP will therefore learn volatility, which is real and useful for "
|
|
"position sizing and drawdown control, but it CANNOT create expectancy: knowing the "
|
|
"next leg spans 3 ATR is worth nothing without knowing which side it spans first. "
|
|
"Agrees with the random-entry exit test (payoff ratio 0.92->5.72, expectancy flat). "
|
|
"Build the head for risk control and stop looking for edge in the exit.");
|
|
else
|
|
Print(ID + ": MI excursion VERDICT - NOTHING clears, INCLUDING the range control. Volatility "
|
|
"clustering is about the most robust regularity in markets, so a range target at the "
|
|
"noise floor points at the measurement, not the market - check the excursion cache "
|
|
"filled and that the sample is not dominated by one volatility regime.");
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ReportFeatureLagProfile(void)
|
|
{
|
|
int maxLag = (int)MathMin(MathMax(m_historyBars, 0), MI_LAG_MAX_PROFILE - 1);
|
|
if(maxLag <= 0)
|
|
return 0;
|
|
//--- Per-lag draws retained for the SAME reason the geometry scan retains its own: this report reads a
|
|
//--- profile of ~20 lags, so "does lag k clear ITS OWN null" is the wrong question at every k. See the
|
|
//--- family-wise block below.
|
|
double lagDraws[MI_LAG_MAX_PROFILE][MI_LAG_PERMUTATIONS];
|
|
double lagExcess[MI_LAG_MAX_PROFILE];
|
|
int lagCount[MI_LAG_MAX_PROFILE];
|
|
bool lagValid[MI_LAG_MAX_PROFILE];
|
|
double atZero = 0.0;
|
|
for(int k = 0; k <= maxLag; k++)
|
|
{
|
|
lagValid[k] = false;
|
|
lagExcess[k] = 0.0;
|
|
lagCount[k] = 0;
|
|
double cols[];
|
|
int labels[];
|
|
int n = BuildMiSample(cols, labels, 0, k);
|
|
if(n < MI_MIN_SAMPLES)
|
|
continue;
|
|
double observed = ScoreMiSample(cols, labels, n, false);
|
|
if(observed < 0.0)
|
|
continue;
|
|
//--- ScoreMiSample shuffles IN PLACE, so the observed statistic must be taken first (above) and the
|
|
//--- draws then reuse the same extracted sample - which is what makes this affordable at all.
|
|
double floorSum = 0.0;
|
|
int draws = 0;
|
|
for(int s = 0; s < MI_LAG_PERMUTATIONS; s++)
|
|
{
|
|
double sc = ScoreMiSample(cols, labels, n, true);
|
|
if(sc < 0.0)
|
|
continue;
|
|
floorSum += sc;
|
|
lagDraws[k][draws] = sc;
|
|
draws++;
|
|
}
|
|
if(draws < 2)
|
|
continue;
|
|
lagExcess[k] = observed - (floorSum / draws);
|
|
lagCount[k] = draws;
|
|
lagValid[k] = true;
|
|
if(k == 0)
|
|
atZero = lagExcess[k];
|
|
}
|
|
//--- FAMILY-WISE CORRECTION ACROSS LAGS. The first version of this report tested each lag against its
|
|
//--- own null at alpha=0.05 across ~21 lags, which is one expected false positive per run before any
|
|
//--- signal exists - and correlated features make them arrive in CLUSTERS that read like a hump. It
|
|
//--- did exactly that on SP500 H1: 2026-08-06 13:55 starred nothing, 16:22 starred k6/k10/k12/k16 and
|
|
//--- concluded "information survives to lag 16" - same instrument, same 31 features, same 2009 samples,
|
|
//--- while the headline MI moved the other way (p 0.4478 -> 0.8756, observed BELOW its null mean).
|
|
//--- Non-replication on identical data is the signature of an uncorrected multiple comparison.
|
|
//---
|
|
//--- So the bar is the null OF THE MAXIMUM over lags, exactly as the barrier-geometry winner test does
|
|
//--- over candidates: one draw from every lag, keep the largest, repeat. A lag clears only by beating
|
|
//--- that. Draws are centred leave-one-out so each is centred by a mean excluding itself, matching how
|
|
//--- the observed excess is centred. Independence across lags overstates the spread of the maximum
|
|
//--- (neighbouring lags share nearly all their feature window), so this errs toward rejecting.
|
|
int fwDraws = MI_LAG_PERMUTATIONS;
|
|
int validLags = 0;
|
|
for(int k = 0; k <= maxLag; k++)
|
|
if(lagValid[k])
|
|
{
|
|
fwDraws = (int)MathMin(fwDraws, lagCount[k]);
|
|
validLags++;
|
|
}
|
|
double fwMax[MI_LAG_PERMUTATIONS];
|
|
if(validLags <= 0)
|
|
fwDraws = 0;
|
|
for(int s = 0; s < fwDraws; s++)
|
|
{
|
|
double worst = -DBL_MAX;
|
|
for(int k = 0; k <= maxLag; k++)
|
|
{
|
|
if(!lagValid[k])
|
|
continue;
|
|
double sum = 0.0;
|
|
for(int q = 0; q < lagCount[k]; q++)
|
|
sum += lagDraws[k][q];
|
|
double loo = (sum - lagDraws[k][s]) / (lagCount[k] - 1);
|
|
double e = lagDraws[k][s] - loo;
|
|
if(e > worst)
|
|
worst = e;
|
|
}
|
|
fwMax[s] = worst;
|
|
}
|
|
string profile = "";
|
|
int deepest = 0;
|
|
for(int k = 0; k <= maxLag; k++)
|
|
{
|
|
if(!lagValid[k])
|
|
{
|
|
profile += StringFormat(" k%d=n/a", k);
|
|
continue;
|
|
}
|
|
int atLeast = 0;
|
|
for(int s = 0; s < fwDraws; s++)
|
|
if(fwMax[s] >= lagExcess[k])
|
|
atLeast++;
|
|
double pFw = (fwDraws > 0) ? (double)(1 + atLeast) / (fwDraws + 1) : 1.0;
|
|
bool clears = (fwDraws > 0 && pFw <= MI_LAG_ALPHA);
|
|
if(clears)
|
|
deepest = k;
|
|
profile += StringFormat(" k%d=%+.5f%s", k, lagExcess[k], (clears ? "*" : ""));
|
|
}
|
|
Print(ID + StringFormat(": MI feature-lag profile (features from bar i+k, LABEL PINNED to the entry "
|
|
"bar i, so every k is causal; value is excess over that lag's own "
|
|
"block-permutation null; '*' = p<=%.2f against the null of the MAXIMUM over "
|
|
"%d lags, not against the lag's own null - %d lags tested one at a time would "
|
|
"star one per run on noise alone) -%s",
|
|
MI_LAG_ALPHA, validLags, validLags, profile));
|
|
if(deepest <= 0)
|
|
Print(ID + StringFormat(": MI feature-lag profile - NOTHING clears the family-wise null at ANY lag "
|
|
"out to %d bars (entry bar itself %+.5f). The %d-bar lookback is not costing "
|
|
"us information; there is none to lose. This is the blind spot the earlier "
|
|
"reports had: they scored the entry bar alone, so they could not have "
|
|
"distinguished 'no signal anywhere' from 'signal only in the older bars'.",
|
|
maxLag, atZero, maxLag));
|
|
else
|
|
Print(ID + StringFormat(": MI feature-lag profile - information survives to lag %d of %d, clearing "
|
|
"the null of the maximum over %d lags. A lookback shorter than %d would "
|
|
"discard measurable information; a longer one adds input width for none. "
|
|
"BEFORE ACTING ON THIS: re-run it. An uncorrected version of this report "
|
|
"gave opposite answers on two runs over identical data, so one run is not "
|
|
"a result - the shape has to reappear, and ideally on a second instrument.",
|
|
deepest, maxLag, validLags, deepest + 1));
|
|
return deepest;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ReportBarrierGeometryScan(void)
|
|
{
|
|
//--- SL x1 is deliberately absent: MIN_SL_ATR_MULTIPLIER floors it anyway, and it was rejected on this
|
|
//--- instrument as too tight to survive normal noise. TP grid is exactly the TAKE_PROFIT_MODE enum.
|
|
double slGrid[] = { 2.0, 3.0 };
|
|
int tpGrid[] = { 2, 3, 4, 6, 8, 10 };
|
|
int savedHorizon = m_barrierHorizonBars;
|
|
int barsNow = m_labelCacheBars;
|
|
uint t0 = GetTickCount();
|
|
string rows = "";
|
|
double bestExcess = -1.0;
|
|
string bestName = "";
|
|
int bestSl = 0, bestTp = 0;
|
|
//--- Per-candidate null draws, retained so the winner can be tested against the null of the MAXIMUM
|
|
//--- rather than against its own. Only ELIGIBLE candidates are enrolled: the family the maximum was
|
|
//--- actually taken over is the family the gate must correct for, and a disqualified pairing can never
|
|
//--- be the winner however it scores.
|
|
double drawMat[MI_GEOMETRY_MAX_CANDIDATES][MI_GEOMETRY_PERMUTATIONS];
|
|
int drawCount[MI_GEOMETRY_MAX_CANDIDATES];
|
|
int candidates = 0;
|
|
double cfgSl = 0.0, cfgTp = 0.0;
|
|
BarrierMultiples(cfgSl, cfgTp);
|
|
double cfgExcess = -1.0;
|
|
m_barrierScanLiveLabels = true;
|
|
for(int a = 0; a < ArraySize(slGrid); a++)
|
|
for(int b = 0; b < ArraySize(tpGrid); b++)
|
|
{
|
|
//--- A target tighter than the stop inverts the trade's whole premise and none of the shipped
|
|
//--- pairings do it; skip rather than rank something nobody can select sensibly.
|
|
if((double)tpGrid[b] < slGrid[a])
|
|
continue;
|
|
m_barrierScanSlMult = slGrid[a];
|
|
m_barrierScanTpMult = (double)tpGrid[b];
|
|
m_barrierHorizonBars = ComputeBarrierHorizonBars(barsNow);
|
|
bool clamped = m_barrierHorizonClamped;
|
|
m_barrierScanTimeouts = 0;
|
|
double gc[];
|
|
int gl[];
|
|
int gn = BuildMiSample(gc, gl);
|
|
if(gn < MI_MIN_SAMPLES)
|
|
continue;
|
|
double obs = ScoreMiSample(gc, gl, gn, false);
|
|
//--- Class shares of THIS geometry's label, so a geometry that scores well by having almost
|
|
//--- nothing left to predict is visible as such instead of winning quietly.
|
|
int cB = 0, cS = 0;
|
|
for(int q = 0; q < gn; q++)
|
|
{
|
|
if(gl[q] == 0)
|
|
cB++;
|
|
else
|
|
if(gl[q] == 1)
|
|
cS++;
|
|
}
|
|
double dirShare = 100.0 * (cB + cS) / gn;
|
|
double timeoutShare = 100.0 * m_barrierScanTimeouts / gn;
|
|
//--- The MIN REWARD:RISK test that used to gate enrolment here is GONE (2026-08-09) along with
|
|
//--- Min_Risk_Reward_Ratio itself. Its purpose was to stop the scan crowning a geometry the live
|
|
//--- rejection filter would then throw every setup away at - but with no rejection filter there
|
|
//--- is nothing to collide with, and excluding low-ratio pairings was excluding them on a rule
|
|
//--- rather than on the measurement. CLAMPING remains disqualifying, and for an unrelated reason
|
|
//--- that still holds: a clamped label describes a trade truncated by the horizon rather than
|
|
//--- resolved at SL/TP, so it is not the target the EA would actually hold to.
|
|
bool eligible = !clamped;
|
|
//--- These draws now serve two purposes. Per candidate they still centre the printed score. Across
|
|
//--- candidates they form the null of the maximum, which is the only thing that can say whether the
|
|
//--- WINNER is real - so they are retained rather than reduced to a mean and discarded.
|
|
double nullSum = 0.0;
|
|
int nd = 0;
|
|
for(int s = 0; s < MI_GEOMETRY_PERMUTATIONS; s++)
|
|
{
|
|
double sc = ScoreMiSample(gc, gl, gn, true);
|
|
if(sc < 0.0)
|
|
continue;
|
|
nullSum += sc;
|
|
if(eligible && candidates < MI_GEOMETRY_MAX_CANDIDATES)
|
|
drawMat[candidates][nd] = sc;
|
|
nd++;
|
|
}
|
|
if(eligible && candidates < MI_GEOMETRY_MAX_CANDIDATES)
|
|
{
|
|
drawCount[candidates] = nd;
|
|
candidates++;
|
|
}
|
|
double nullMean = (nd > 0) ? nullSum / nd : -1.0;
|
|
double excess = (nullMean >= 0.0) ? (obs - nullMean) : 0.0;
|
|
//--- Base rate m/(m+k) IS the break-even win rate at this geometry - print it so the ranking is
|
|
//--- read next to the bar the model would have to clear, not in isolation.
|
|
double breakeven = 100.0 * slGrid[a] / (slGrid[a] + (double)tpGrid[b]);
|
|
//--- Ranking is now purely the measurement: every unclamped pairing competes, whatever its
|
|
//--- reward:risk. Note the break-even printed alongside is what makes a low-ratio winner
|
|
//--- readable rather than alarming - 1:1 needs 50% precision where 1:3 needs 25%, and the
|
|
//--- scan's value column is already excess information over that geometry's OWN null, so the
|
|
//--- comparison across pairings is like-for-like.
|
|
string name = StringFormat("%.0f:%d", slGrid[a], tpGrid[b]);
|
|
rows += StringFormat("%s%s(h%d%s,be%.0f%%,dir%.0f%%,to%.0f%%)=%+.5f", (rows == "" ? "" : " "),
|
|
name, m_barrierHorizonBars, (clamped ? "!" : ""), breakeven,
|
|
dirShare, timeoutShare, excess);
|
|
//--- Only unclamped, tradeable geometries are eligible to WIN - see the header. The rest are
|
|
//--- still printed, so a disqualification is visible rather than a silent omission.
|
|
if(eligible && excess > bestExcess)
|
|
{
|
|
bestExcess = excess;
|
|
bestName = name;
|
|
//--- The grid values ARE the enum values (SL_ATR_x2 == 2, TP_ATR_x8 == 8), so the winning
|
|
//--- pairing can be adopted directly with no lookup table to drift out of step.
|
|
bestSl = (int)slGrid[a];
|
|
bestTp = tpGrid[b];
|
|
}
|
|
if(slGrid[a] == cfgSl && (double)tpGrid[b] == cfgTp)
|
|
cfgExcess = excess;
|
|
}
|
|
m_barrierScanLiveLabels = false;
|
|
m_barrierScanSlMult = 0.0;
|
|
m_barrierScanTpMult = 0.0;
|
|
m_barrierHorizonBars = savedHorizon;
|
|
Print(ID + StringFormat(": barrier-geometry scan (SL:TP; h=horizon, '!'=CLAMPED and disqualified - a clamped "
|
|
"label truncates a trade the EA would hold to SL/TP; be=break-even win rate, "
|
|
"dir=%%bars with a tradeable direction, to=%%timed out; value is entry-time "
|
|
"information in nats above that geometry's own null) - %s | configured "
|
|
"%.0f:%.0f scores %+.5f, best eligible is %s at %+.5f (%.1fs)",
|
|
rows, cfgSl, cfgTp, cfgExcess, (bestName == "" ? "none" : bestName), bestExcess,
|
|
(GetTickCount() - t0) / 1000.0));
|
|
//--- FAMILY-WISE GATE. bestExcess is a MAXIMUM over the eligible candidates, and the maximum of several
|
|
//--- draws from a null sits well above any single draw from it - so testing the winner against its own
|
|
//--- null asks the wrong question and will crown a winner on pure noise almost every time. What follows
|
|
//--- rebuilds the null OF THE MAXIMUM: take one permutation draw from every candidate, keep the largest,
|
|
//--- repeat. bestExcess then has to beat that distribution, not a single-candidate one.
|
|
//---
|
|
//--- The draws are centred LEAVE-ONE-OUT so the comparison is like for like: the observed score is
|
|
//--- centred by draws that do not contain it, so each draw must be too. Centring a draw by a mean that
|
|
//--- includes it shrinks it toward zero, which would deflate the null and let the winner through.
|
|
//---
|
|
//--- Draws are independent across candidates here while the real ones are correlated (the candidates
|
|
//--- share features and heavily overlapping label windows). Independence makes the maximum MORE spread
|
|
//--- out than the truth, so the gate errs toward rejecting - the safe direction when passing costs a
|
|
//--- full relabel and retrain of every topology.
|
|
double pFamily = 1.0;
|
|
int fwDraws = 0;
|
|
if(candidates > 0)
|
|
{
|
|
fwDraws = MI_GEOMETRY_PERMUTATIONS;
|
|
for(int c = 0; c < candidates; c++)
|
|
fwDraws = (int)MathMin(fwDraws, drawCount[c]);
|
|
int atLeast = 0;
|
|
for(int s = 0; s < fwDraws; s++)
|
|
{
|
|
double worst = -DBL_MAX;
|
|
for(int c = 0; c < candidates; c++)
|
|
{
|
|
if(drawCount[c] < 2)
|
|
continue;
|
|
double sum = 0.0;
|
|
for(int q = 0; q < drawCount[c]; q++)
|
|
sum += drawMat[c][q];
|
|
double loo = (sum - drawMat[c][s]) / (drawCount[c] - 1);
|
|
double e = drawMat[c][s] - loo;
|
|
if(e > worst)
|
|
worst = e;
|
|
}
|
|
if(worst > -DBL_MAX && worst >= bestExcess)
|
|
atLeast++;
|
|
}
|
|
pFamily = (fwDraws > 0) ? (double)(1 + atLeast) / (fwDraws + 1) : 1.0;
|
|
}
|
|
bool winnerReal = (bestName != "" && bestExcess > 0.0 && fwDraws > 0 && pFamily <= MI_GEOMETRY_ALPHA);
|
|
Print(ID + StringFormat(": barrier-geometry winner test - %s at %+.5f is the best of %d ELIGIBLE "
|
|
"candidates, so it is tested against the null of the maximum over %d, not its "
|
|
"own: p=%.4f over %d draws (need <=%.2f). %s", (bestName == "" ? "none" : bestName),
|
|
bestExcess, candidates, candidates, pFamily, fwDraws, MI_GEOMETRY_ALPHA,
|
|
(winnerReal ? "CLEARS - the ranking is not selection noise."
|
|
: "DOES NOT CLEAR - a max this large happens routinely when every candidate is "
|
|
"pure noise, so the ranking carries no information and the top row is not a "
|
|
"finding. Change nothing.")));
|
|
//--- ADOPT, don't advise. SL_Mode/TP_Mode stopped being inputs on 2026-08-07, so this scan is now the
|
|
//--- thing that chooses the barrier - which is exactly why the family-wise gate above had to exist
|
|
//--- first. Three conditions, all necessary:
|
|
//--- winnerReal - it beat the null of the MAXIMUM, not merely the incumbent and not merely zero.
|
|
//--- m_eraCount==0 - relabelling a partly-trained net would move the target out from under weights
|
|
//--- already fitted to the old one. Same gate the indicator tuner uses.
|
|
//--- != current - nothing to do when the measurement agrees with the default.
|
|
//--- A model that already exists never reaches here with anything to change: its geometry is pinned in
|
|
//--- the .cfg and adopted at load, so the pairing a run trains on is the pairing it keeps.
|
|
if(winnerReal && m_eraCount == 0 && bestSl > 0 && bestTp > 0
|
|
&& (bestSl != m_sl_mode || bestTp != m_tp_mode))
|
|
{
|
|
Print(ID + StringFormat(": adopting barrier geometry %s - it carries %+.5f nats of entry-time "
|
|
"information against the configured %.0f:%.0f's %+.5f, and cleared the "
|
|
"family-wise gate. Relabelling and training on it. Chance precision equals "
|
|
"break-even at EVERY geometry, so this does not hand us expectancy; it puts "
|
|
"more of the answer inside the features' reach, which is the one thing no "
|
|
"change of topology can do.", bestName, bestExcess, cfgSl, cfgTp, cfgExcess));
|
|
m_sl_mode = bestSl;
|
|
m_tp_mode = bestTp;
|
|
//--- The cache holds labels computed under the OLD barriers, so it has to be discarded rather than
|
|
//--- appended to - Train()'s !m_labelCachePrebuilt gate then rebuilds it under the adopted pair
|
|
//--- before era 0 starts.
|
|
m_labelCachePrebuilt = false;
|
|
ArrayInitialize(m_labelCacheHasValue, false);
|
|
//--- AND UNLATCH THE HORIZON, which is otherwise resolved once per process and held. Adopting a
|
|
//--- wider target without this labels the new geometry against the OLD ceiling - 2:8 wants ~192
|
|
//--- bars and would silently get 2:6's 128 - which is precisely the truncation that made every
|
|
//--- model learn "target within 128 bars" while the EA holds to SL/TP (fixed 2026-08-01 in
|
|
//--- 168422f). The truncation lands in Neutral, not in the timeout counter that watches for it, so
|
|
//--- it does not announce itself. EnsureBarrierHorizon() re-derives and re-logs on the next call.
|
|
m_barrierHorizonResolved = false;
|
|
}
|
|
else
|
|
if(winnerReal && m_eraCount > 0 && bestSl > 0 && (bestSl != m_sl_mode || bestTp != m_tp_mode))
|
|
Print(ID + ": barrier-geometry scan prefers " + bestName + ", but this model is already trained "
|
|
"(era " + IntegerToString(m_eraCount) + "). Its geometry is pinned to what it learned; "
|
|
"delete the weights if you want it re-measured.");
|
|
else
|
|
if(bestName == "")
|
|
Print(ID + ": barrier-geometry scan - every geometry with a long enough horizon was "
|
|
"disqualified or scored at zero. Nothing here to switch to; the limit is the feature "
|
|
"set, not the target.");
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Outer loop around Train(). Tuning is now a one-shot filter pass |
|
|
//| that runs BEFORE the first era and costs seconds, so this is a |
|
|
//| straight pass-through to Train() on every later call. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::TuneIndicatorsAndTrain(datetime StartTrainBar = 0)
|
|
{
|
|
//--- Publish the caller's window anchor so StartLabelCachePrebuild() sizes its window with the SAME
|
|
//--- expression Train() uses. This assignment existed before the God-class split and was dropped by
|
|
//--- it, leaving m_tuneStartTrainBar permanently 0 - harmless while nothing read it, and a real defect
|
|
//--- the moment 0c85c54 made the prebuild reset dtStudied from it: Train() then computed
|
|
//--- max(StartTrainBar, windowFloor) while the prebuild computed max(0, windowFloor), so the two
|
|
//--- disagreed about `bars`, EnsureBarCachesCapacity() saw a changed size every era start, and each
|
|
//--- era immediately re-armed a full 38k-bar prebuild instead of training.
|
|
m_tuneStartTrainBar = StartTrainBar;
|
|
bool anyTunable = (m_useADCumulativeDelta || m_useADShorteningOfThrust || m_useADWyckoffEventStream ||
|
|
m_useADWyckoffFailedStructure || m_useADWyckoffSignificantBarInversion ||
|
|
m_useMA || m_useRSI || m_useMACD || m_useIchimoku);
|
|
//--- Tune once per fresh model, before any weight has been trained. Gated on m_labelCachePrebuilt
|
|
//--- because the score needs labels, and on era 0 because re-tuning a partly-trained network would
|
|
//--- change its inputs out from under weights already fitted to the old ones.
|
|
if(m_autoTuneIndicators && anyTunable && !m_tuneFilterDone && m_labelCachePrebuilt && m_eraCount == 0)
|
|
{
|
|
m_tuneFilterDone = true;
|
|
SetStatusLabel(ID + " : scoring indicator settings...");
|
|
TuneIndicatorsByFilter();
|
|
//--- the winning parameters change the input vector, so the network must start from scratch on it
|
|
BuildFreshTopology();
|
|
}
|
|
//--- The DIAGNOSTIC half runs even when the sweep does not: on a resumed model, on one whose tuner is
|
|
//--- switched off, and on one with nothing tunable. It reads the cached features and writes nothing,
|
|
//--- so none of the reasons the sweep is gated apply to it - and tying it to that gate meant the only
|
|
//--- way to see the answer on a running model was to delete the model.
|
|
//---
|
|
//--- THAT INTENT WAS NOT ACHIEVED UNTIL 2026-08-07. Moving the diagnostic out of the tuner's gate
|
|
//--- left it behind m_labelCachePrebuilt, which has exactly the same effect: the eager label pre-scan
|
|
//--- runs only on a FRESH start, because a resumed net labels lazily per bar (see the "skipped
|
|
//--- entirely when a trained net was loaded from disk" note in the prebuild). So on a resumed model
|
|
//--- the flag is false forever and the entire MI block - headline, positive control, alignment scan,
|
|
//--- lag profile, geometry scan, winner test - silently never ran. Measured on SP500 H1 2026-08-07:
|
|
//--- attached at era 271, still nothing by era 314, and every diagnostic captured on 08-05/06 came
|
|
//--- immediately after a weights reset. The only way to see the answer was still to delete the model.
|
|
//---
|
|
//--- So drive the prebuild ourselves when it is the only thing missing. It is safe on a trained net:
|
|
//--- its one fresh-net side effect, pushing the output-layer bias toward the dominant class, is
|
|
//--- already gated on m_eraCount == 0, and the scan itself only fills label caches. Train()'s own
|
|
//--- m_labelPrebuildActive gate advances it to completion, so this costs one short deferral (~1s at
|
|
//--- 38k bars) on the first attach and nothing afterwards.
|
|
//---
|
|
//--- NOT sampled from the lazily-filled cache instead: BuildMiSample skips bars that carry no cached
|
|
//--- label, so on a resumed model it would quietly score whichever subset training happened to have
|
|
//--- visited. That is a biased subsample presented as a measurement - the failure mode this whole
|
|
//--- diagnostic exists to catch.
|
|
else if(!m_miReportDone && !m_labelCachePrebuilt && !m_labelPrebuildActive)
|
|
{
|
|
//--- Announce only on a start that actually took. StartLabelCachePrebuild() returns without arming
|
|
//--- if the buffers/history are not ready yet and is simply retried on the next call, so printing
|
|
//--- unconditionally would repeat the line once per bar event until it succeeds.
|
|
StartLabelCachePrebuild();
|
|
//--- Says WHICH case this is rather than asserting the resumed one. The first version claimed
|
|
//--- "resumed from disk" unconditionally, and then printed it above a "seeding era 0" line on a
|
|
//--- brand-new model - the branch fires whenever the cache is not built, which is equally true
|
|
//--- before a fresh model's first prebuild. A diagnostic that misreports its own trigger is worse
|
|
//--- than one that says nothing, because it gets quoted back as evidence.
|
|
if(m_labelPrebuildActive)
|
|
Print(ID + (m_modelLoadedFromDisk
|
|
? ": MI diagnostics need a complete label cache and this model resumed from disk "
|
|
"(labels are filled lazily, so the cache covers only the bars training has "
|
|
"visited) - running the one-time pre-scan now, then the report. Training resumes "
|
|
"where it left off."
|
|
: ": MI diagnostics need a complete label cache and this model has not built one yet "
|
|
"- running the pre-scan now, then the report."));
|
|
}
|
|
else if(!m_miReportDone && m_labelCachePrebuilt)
|
|
{
|
|
//--- WAIT FOR THE CROSS-ASSET PANEL. It is part of the feature vector but it is built inside
|
|
//--- Train(), so on a fresh run this diagnostic would otherwise describe a NARROWER vector than
|
|
//--- the one training goes on to use. Observed 2026-08-02 on SP500 H1: the MI report, the
|
|
//--- alignment scan and the barrier-geometry scan all ran at 00:41:25, while the panel first
|
|
//--- built successfully at 01:12:47 - so every number they printed, including the geometry scan
|
|
//--- that is supposed to CHOOSE the training target, was measured on a feature set training
|
|
//--- never saw. Train() rebuilds the panel each era, so simply deferring lands the report on an
|
|
//--- era where the vector is complete.
|
|
//--- Never wait forever: a terminal that cannot sync the reference symbols (the tester loads
|
|
//--- auxiliary symbols from the terminal, not the server) must still get its diagnostics, with
|
|
//--- the gap stated rather than hidden.
|
|
if(m_crossAsset.IsReady() || m_miReportDeferrals >= MI_REPORT_MAX_DEFERRALS)
|
|
ReportFeatureLabelInformation();
|
|
else
|
|
m_miReportDeferrals++;
|
|
}
|
|
Train(StartTrainBar);
|
|
}
|
|
#endif // WARRIOR_AIBASE_AUTOTUNE_MQH
|