forked from mnbvc188199/Warrior_EA
f102a695d5
MEASURED 2026-08-17 19:06 on USDJPY, in the fresh run: 19:06:38 LSTM adopting barrier geometry 2:10 ... geometry authority 19:06:40 LSTM triple-barrier labels - stop 2.00 target 10.00, horizon 256 19:06:44 PAI / CONV / HYB break-even 33.3%, mean label lifespan 19.2 bars 19:06:45 LSTM break-even 16.7%, mean label lifespan 81.4 bars One chart, four members, two targets. A "Buy" from LSTM meant "10 ATR before a 2 ATR stop within 256 bars"; a "Buy" from PAI meant "3.21 before 1.61 within 64". The orchestrator averages those votes and the joint gate certifies the average as though they answered one question. And g_DerivedSlAtrMult - which places the LIVE order - is a single global, so the stop actually sent was whichever member wrote last: the same last-writer-wins class of bug as the live-exit confidence. CAUSE, and it is mine. The geometry scan sits at the end of the MI chain, and that chain runs ONCE PER CHART (g_ensembleChartMiReportDone) - whichever member reaches it first measures and the rest skip. Harmless while the scan only PRINTED;62a719fmade it authoritative and turned a skipped report into a skipped DECISION. The indicator tuner already had this doctrine (g_ensembleChartTuneSettings); the geometry had no equivalent. - g_ensembleChartGeomAdopted/Sl/Tp/SlMode/TpMode: the donor publishes its pairing, the siblings adopt it in the MI-skip branch. Ordering is safe by construction - MQL5 is single-threaded per chart and the donor sets g_ensembleChartMiReportDone only after the chain (and so the adoption) returns, so any member taking the skip branch does so strictly afterwards. - ApplyAdoptedGeometry(): the eleven side effects an adopted pairing must carry - derived pair, legacy mode ints, g_Derived* live globals, .cfg rewrite, label cache invalidation, horizon unlatch - in ONE function, because there are now two callers and duplicating them is how the two paths drift. - Guarded on era 0 for the donor's own reason: relabelling a partly trained net moves the target out from under weights already fitted to the old one. STILL OPEN: dead MA handles were not eliminated bycb30360. They now appear at a different site (SP500 19:06:35, during "label prebuild", and on PAI - the member that RAN the sweep), so there is a second handle-churn path I have not found. Recovery works and the sharing diagnosis stands; the trigger is not only the tuner's adopt branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2093 lines
130 KiB
MQL5
2093 lines
130 KiB
MQL5
//+------------------------------------------------------------------+
|
|
//| Warrior_EA |
|
|
//| AnimateDread |
|
|
//| |
|
|
//| Filter-based indicator auto-tuner (mutual information scoring). |
|
|
//| |
|
|
//| PARTIAL IMPLEMENTATION FILE - not standalone. |
|
|
//| This holds CExpertSignalAIBase method BODIES only. The class |
|
|
//| declaration lives in Expert\ExpertSignalAIBase.mqh, which |
|
|
//| #includes this file at the bottom, after the declaration. Do not |
|
|
//| include it anywhere else and do not compile it on its own. |
|
|
//| |
|
|
//| Split out purely to make the 8216-line original navigable; the |
|
|
//| code inside was moved verbatim, not rewritten. |
|
|
//+------------------------------------------------------------------+
|
|
#ifndef WARRIOR_AIBASE_AUTOTUNE_MQH
|
|
#define WARRIOR_AIBASE_AUTOTUNE_MQH
|
|
//--- ONCE-PER-CHART gate for the MI diagnostic suite on an AI_HYBRID ensemble. The suite (headline
|
|
//--- MI, positive control, alignment scan, lag profile, barrier-geometry scan + winner test) is a
|
|
//--- pure MEASUREMENT of this chart's features and labels - identical inputs for every ensemble
|
|
//--- member, so running it four times quadruples the slowest part of "getting ready" (the permuted
|
|
//--- nulls are ~200 draws per line) to report the same numbers four times. The first member to reach
|
|
//--- it runs it; the rest log one line and skip. CAVEAT, documented not hidden: the geometry scan can
|
|
//--- in principle ADOPT its winner under the family-wise gate (it never has on any instrument -
|
|
//--- "Change nothing" every run); if it ever clears, this gate makes that adoption donor-only and
|
|
//--- must be revisited. Solo charts are untouched.
|
|
bool g_ensembleChartMiReportDone = false;
|
|
//--- ONCE-PER-CHART share of the indicator auto-tune SWEEP on an AI_HYBRID ensemble, same doctrine as
|
|
//--- the MI gate above: the sweep scores candidate indicator settings by feature/label MI, and every
|
|
//--- ensemble member holds identical indicators, identical cached features and identical labels, so all
|
|
//--- N sweeps are the same deterministic calculation (verified 2026-08-16 on SP500 H4: four members,
|
|
//--- byte-identical scores, spans and selection p). Worse, the sweep ends in the full MI diagnostic
|
|
//--- suite (ReportFeatureLabelInformation at its tail), which the MI gate above never intercepts on the
|
|
//--- sweep path - so each duplicate sweep also duplicated the ~200-draw permutation nulls, the slowest
|
|
//--- single block of "getting ready". The first member runs the sweep and publishes its outcome here;
|
|
//--- the rest apply the outcome (install the winner, or keep the configured settings the sweep restored)
|
|
//--- and skip both the sweep and the report. Same caveat as the MI gate: any winner ADOPTION is made by
|
|
//--- the donor and applied to every member via the flattened settings below, which is the consistent
|
|
//--- choice - members training on divergent feature vectors would not be an ensemble. Solo charts are
|
|
//--- untouched.
|
|
bool g_ensembleChartTuneDone = false;
|
|
bool g_ensembleChartTuneInstalled = false; // did the donor's sweep clear the family-wise gate and install?
|
|
double g_ensembleChartTuneSettings[]; // CADIndicatorTuner::Flatten() of the donor's final settings
|
|
//--- THE SAME DOCTRINE, APPLIED TO THE BARRIER GEOMETRY - and it was missing, which broke the ensemble.
|
|
//--- The geometry scan lives at the end of the MI chain, and that chain runs ONCE PER CHART
|
|
//--- (g_ensembleChartMiReportDone): whichever member reaches it first measures, and the other three skip
|
|
//--- the whole thing. That was harmless while the scan only PRINTED. Once it became authoritative
|
|
//--- (62a719f) the donor started relabelling itself and nobody else, so a single chart's members trained
|
|
//--- on DIFFERENT TARGETS - and the orchestrator averages their votes and the joint gate certifies the
|
|
//--- combined vote as if they answered one question.
|
|
//--- MEASURED 2026-08-17 19:06 on USDJPY: LSTM adopted 2:10 (horizon 256, break-even 16.7%, mean label
|
|
//--- lifespan 81.4 bars) while PAI/CONV/HYB stayed at 1.61:3.21 (horizon 64, break-even 33.3%, lifespan
|
|
//--- 19.2). A "Buy" from LSTM and a "Buy" from PAI were not the same claim, and averaging them is not a
|
|
//--- vote. g_DerivedSlAtrMult - which places the LIVE order - is one global, so the stop actually sent
|
|
//--- was whichever member wrote last.
|
|
bool g_ensembleChartGeomAdopted = false; // did the donor's scan adopt a pairing the siblings must take?
|
|
double g_ensembleChartGeomSl = 0.0; // the DERIVED pair (the one authority - see BarrierMultiples)
|
|
double g_ensembleChartGeomTp = 0.0;
|
|
int g_ensembleChartGeomSlMode = 0; // legacy mode ints, kept in step for the fallback/fingerprint
|
|
int g_ensembleChartGeomTpMode = 0;
|
|
#ifdef WARRIOR_EXPORT_FEATURES
|
|
//+------------------------------------------------------------------+
|
|
//| RESEARCH BUILD ONLY - see the declaration comment. |
|
|
//| |
|
|
//| Every research question so far has cost a compile, a deploy, an |
|
|
//| attach and a log read - minutes each, and the answer arrives one |
|
|
//| hypothesis at a time. That loop, not the modelling, is what has |
|
|
//| made this slow. Exporting the feature matrix ONCE moves the whole |
|
|
//| question offline, where a hypothesis costs seconds and real tools |
|
|
//| (joint mutual information, gradient boosting, proper walk-forward |
|
|
//| cross-validation) are available - none of which can be written in |
|
|
//| MQL5 in reasonable time. |
|
|
//| |
|
|
//| Exports the RAW BARS next to the features deliberately: with OHLC |
|
|
//| and ATR offline, every barrier geometry, every horizon and every |
|
|
//| in-trade target can be recomputed without touching MetaTrader |
|
|
//| again. The bar TIME goes out too, which makes session, hour and |
|
|
//| day-of-week features derivable for free - and those are the only |
|
|
//| inputs in play that are NOT a transform of the same OHLCV series. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ExportFeatureMatrix(void)
|
|
{
|
|
if(MQLInfoInteger(MQL_OPTIMIZATION))
|
|
return;
|
|
int barsNow = Bars(m_symbol.Name(), PERIOD_CURRENT);
|
|
//--- Clamp BEFORE the emptiness test, so a fully-capped symbol reports the depth it can actually
|
|
//--- export rather than the price-series depth it cannot. Exporting the uncapped count is how this
|
|
//--- path used to fail: ResizeBuffers/RefreshData "succeed" (Refresh() is void), then every MA read
|
|
//--- comes back EMPTY_VALUE and the CSV is written with a dead feature block - a research artefact
|
|
//--- that looks complete and is silently wrong, which is the whole failure class this guards.
|
|
barsNow = ServableBars(barsNow, "feature export");
|
|
if(barsNow <= m_historyBars + 2)
|
|
{
|
|
Print(ID + ": EXPORT - only " + IntegerToString(barsNow) + " bars available, nothing to write");
|
|
return;
|
|
}
|
|
if(!ResizeBuffers(barsNow) || !RefreshData())
|
|
{
|
|
Print(ID + ": EXPORT - buffers not ready (" + IntegerToString(barsNow) + " bars), aborting");
|
|
return;
|
|
}
|
|
EnsureBarCachesCapacity(barsNow);
|
|
EnsureBarrierHorizon(barsNow);
|
|
string dir = eaName + "\\Research\\";
|
|
string fn = dir + m_symbol.Name() + "_" + IntegerToString(_Period) + "_features.csv";
|
|
int h = FileOpen(fn, FILE_COMMON | FILE_WRITE | FILE_CSV | FILE_ANSI, ',');
|
|
if(h == INVALID_HANDLE)
|
|
{
|
|
Print(ID + ": EXPORT - cannot open " + fn + ", error " + IntegerToString(GetLastError()));
|
|
return;
|
|
}
|
|
string header = "idx,time,open,high,low,close,atr";
|
|
for(int f = 0; f < m_neuronsCount; f++)
|
|
header += ",f" + IntegerToString(f);
|
|
FileWrite(h, header);
|
|
//--- Oldest first. The loop walks DOWN the series index, which is forward in time (higher index =
|
|
//--- older), so the file reads chronologically and Python can treat row order as time order.
|
|
int written = 0, skipped = 0;
|
|
uint t0 = GetTickCount();
|
|
for(int i = barsNow - 1; i >= 0; i--)
|
|
{
|
|
TempData.Clear();
|
|
if(!BufferTempData(i) || TempData.Total() < m_neuronsCount)
|
|
{
|
|
skipped++;
|
|
continue;
|
|
}
|
|
double atr = m_ATR.Main(i);
|
|
string row = IntegerToString(i) + "," + IntegerToString((long)m_Time.GetData(i)) + "," +
|
|
DoubleToString(m_Open.GetData(i), _Digits) + "," +
|
|
DoubleToString(m_High.GetData(i), _Digits) + "," +
|
|
DoubleToString(m_Low.GetData(i), _Digits) + "," +
|
|
DoubleToString(m_Close.GetData(i), _Digits) + "," +
|
|
DoubleToString(MathIsValidNumber(atr) ? atr : 0.0, _Digits);
|
|
for(int f = 0; f < m_neuronsCount; f++)
|
|
row += "," + DoubleToString(TempData.At(f), 8);
|
|
FileWrite(h, row);
|
|
written++;
|
|
}
|
|
TempData.Clear();
|
|
FileClose(h);
|
|
Print(ID + StringFormat(": EXPORT COMPLETE - %d rows x %d features -> Common\\Files\\%s "
|
|
"(%d bars skipped for missing features, %.1fs, horizon %d, spread %d points)",
|
|
written, m_neuronsCount, fn, skipped, (GetTickCount() - t0) / 1000.0,
|
|
m_barrierHorizonBars, (int)m_symbol.Spread()));
|
|
ExportRawRates();
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| RESEARCH BUILD ONLY. Raw OHLCV for a GRID of symbols/timeframes, |
|
|
//| not just this chart's. |
|
|
//| |
|
|
//| The 26 engineered features can only be produced for the chart the |
|
|
//| EA is attached to - the indicator handles are bound to |
|
|
//| PERIOD_CURRENT. Raw rates are not: CopyRates serves any symbol |
|
|
//| and any timeframe from a single chart. So one attach yields the |
|
|
//| whole research grid, and every question that does not require the |
|
|
//| EXISTING feature set - a different horizon, a different barrier, |
|
|
//| session/time-of-day effects, features this EA does not have yet - |
|
|
//| can then be answered offline without MetaTrader in the loop at |
|
|
//| all. That is what turns a per-hypothesis cost of minutes into |
|
|
//| seconds, which has been the real bottleneck all along. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ExportRawRates(void)
|
|
{
|
|
string symbols[] = { "SP500", "USDJPY", "XAUUSD", "EURUSD", "GBPUSD", "US30", "NAS100", "BTCUSD" };
|
|
ENUM_TIMEFRAMES tfs[] = { PERIOD_M5, PERIOD_M15, PERIOD_H1, PERIOD_H4, PERIOD_D1 };
|
|
string dir = eaName + "\\Research\\";
|
|
int cells = 0, rowsTotal = 0;
|
|
for(int s = 0; s < ArraySize(symbols); s++)
|
|
{
|
|
//--- Skip silently rather than warn: the grid is deliberately broader than any one broker's symbol
|
|
//--- list, so an absent instrument is expected, not an error.
|
|
if(!SymbolSelect(symbols[s], true))
|
|
continue;
|
|
for(int p = 0; p < ArraySize(tfs); p++)
|
|
{
|
|
MqlRates r[];
|
|
ArraySetAsSeries(r, false); // oldest first, so file order is time order
|
|
int got = CopyRates(symbols[s], tfs[p], 0, 200000, r);
|
|
if(got <= 100)
|
|
continue;
|
|
string fn = dir + symbols[s] + "_" + IntegerToString((int)tfs[p]) + "_rates.csv";
|
|
int h = FileOpen(fn, FILE_COMMON | FILE_WRITE | FILE_CSV | FILE_ANSI, ',');
|
|
if(h == INVALID_HANDLE)
|
|
continue;
|
|
int dg = (int)SymbolInfoInteger(symbols[s], SYMBOL_DIGITS);
|
|
FileWrite(h, "time,open,high,low,close,tickvol,spread");
|
|
for(int i = 0; i < got; i++)
|
|
FileWrite(h, IntegerToString((long)r[i].time) + "," +
|
|
DoubleToString(r[i].open, dg) + "," + DoubleToString(r[i].high, dg) + "," +
|
|
DoubleToString(r[i].low, dg) + "," + DoubleToString(r[i].close, dg) + "," +
|
|
IntegerToString((long)r[i].tick_volume) + "," + IntegerToString(r[i].spread));
|
|
FileClose(h);
|
|
cells++;
|
|
rowsTotal += got;
|
|
Print(ID + StringFormat(": EXPORT rates - %s %s: %d bars", symbols[s],
|
|
EnumToString(tfs[p]), got));
|
|
}
|
|
}
|
|
Print(ID + StringFormat(": EXPORT RATES COMPLETE - %d cells, %d bars total, under Common\\Files\\%s",
|
|
cells, rowsTotal, dir));
|
|
}
|
|
#endif
|
|
//--- The genetic + successive-halving helpers that used to live here (GaRungEras, GaExtract, GaStore,
|
|
//--- GaMutate, GaRandomCandidate, GaBlockCrossover, GaSortAliveByScoreDesc, GaBreedNextGeneration) were
|
|
//--- deleted on 2026-08-01 together with the search they served. See TuneIndicatorsByFilter() below for
|
|
//--- the measured cost that retired them and what replaced it.
|
|
//+------------------------------------------------------------------+
|
|
//| MUTUAL INFORMATION between one cached feature column and the |
|
|
//| triple-barrier label, in nats, over a sample of in-sample bars. |
|
|
//| |
|
|
//| I(X;Y) = sum p(x,y) log( p(x,y) / (p(x) p(y)) ), with the feature |
|
|
//| discretised into MI_BINS EQUAL-FREQUENCY bins. Equal-frequency |
|
|
//| rather than equal-width because these features are ATR-normalised |
|
|
//| and heavy-tailed: fixed-width bins put nearly everything in one |
|
|
//| bucket and report ~0 information for a genuinely useful feature. |
|
|
//| |
|
|
//| Rank-based binning gives equal frequency for free - sort a copy of |
|
|
//| the column, then a value's bin is its rank scaled into MI_BINS. |
|
|
//+------------------------------------------------------------------+
|
|
double CExpertSignalAIBase::FeatureColumnMI(const double &vals[], const int &labels[], int n)
|
|
{
|
|
if(n < MI_MIN_SAMPLES)
|
|
return 0.0;
|
|
double sorted[];
|
|
ArrayResize(sorted, n);
|
|
ArrayCopy(sorted, vals, 0, 0, n);
|
|
ArraySort(sorted);
|
|
//--- A column that never varies carries no information; short-circuit so the log below is never
|
|
//--- reached with a degenerate single-bin histogram.
|
|
if(sorted[0] == sorted[n - 1])
|
|
return 0.0;
|
|
int joint[]; ArrayResize(joint, MI_BINS * 3); ArrayInitialize(joint, 0);
|
|
int px[]; ArrayResize(px, MI_BINS); ArrayInitialize(px, 0);
|
|
int py[]; ArrayResize(py, 3); ArrayInitialize(py, 0);
|
|
for(int i = 0; i < n; i++)
|
|
{
|
|
//--- rank via binary search on the sorted copy; ties land in the same bin, which is correct
|
|
int lo = 0, hi = n - 1, rank = 0;
|
|
while(lo <= hi)
|
|
{
|
|
int mid = (lo + hi) / 2;
|
|
if(sorted[mid] < vals[i])
|
|
{
|
|
rank = mid + 1;
|
|
lo = mid + 1;
|
|
}
|
|
else
|
|
hi = mid - 1;
|
|
}
|
|
int bx = (int)((double)rank * MI_BINS / n);
|
|
if(bx >= MI_BINS)
|
|
bx = MI_BINS - 1;
|
|
int by = labels[i];
|
|
if(by < 0 || by > 2)
|
|
continue;
|
|
joint[bx * 3 + by]++;
|
|
px[bx]++;
|
|
py[by]++;
|
|
}
|
|
double mi = 0.0;
|
|
for(int b = 0; b < MI_BINS; b++)
|
|
{
|
|
if(px[b] <= 0)
|
|
continue;
|
|
for(int c = 0; c < 3; c++)
|
|
{
|
|
int j = joint[b * 3 + c];
|
|
if(j <= 0 || py[c] <= 0)
|
|
continue;
|
|
double pxy = (double)j / n;
|
|
mi += pxy * MathLog(pxy / (((double)px[b] / n) * ((double)py[c] / n)));
|
|
}
|
|
}
|
|
return (mi > 0.0) ? mi : 0.0;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Scores the CURRENT indicator parameters by how much the resulting |
|
|
//| feature vector tells us about the label - the mean per-column |
|
|
//| mutual information over a stratified sample of in-sample bars. |
|
|
//| |
|
|
//| Deliberately scores EVERY column, not just the ones belonging to |
|
|
//| the parameter being swept. Columns the sweep did not touch |
|
|
//| contribute the SAME amount to every candidate, so they shift the |
|
|
//| mean by a constant and cannot change which candidate wins - while |
|
|
//| avoiding any need for this code to know the feature-vector layout, |
|
|
//| which is exactly the kind of coupling that rots. |
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::BuildMiSample(double &cols[], int &labels[], int labelBarOffset = 0,
|
|
int featureBarOffset = 0, int target = MI_TARGET_BARRIER)
|
|
{
|
|
//--- Continuous targets are collected raw here and discretised after the loop, because equal-frequency
|
|
//--- binning needs the whole sample's distribution before any one row can be assigned a bin.
|
|
double raw[];
|
|
bool continuousTarget = (target != MI_TARGET_BARRIER);
|
|
int bars = m_labelCacheBars;
|
|
if(bars <= 0 || m_neuronsCount <= 0)
|
|
return -1;
|
|
//--- Sample the IS region only. The OOS window must not influence which indicator settings ship, or
|
|
//--- the holdout has been used for selection and stops being a holdout at all.
|
|
int oosCutoff = (int)(MathMax(0, MathMin(100, m_oosSplitPct)) / 100.0
|
|
* MathMax(bars - MathMax(m_historyBars, 0), 0));
|
|
int lo = MathMax(oosCutoff, MathMax(m_barrierHorizonBars, 1) + 1);
|
|
int hi = bars - MathMax(m_historyBars, 0) - 1;
|
|
//--- Keep the OFFSET label lookup inside the same bounds as the features, so a shifted scan measures a
|
|
//--- shift and not an edge effect. Widened symmetrically rather than clamping per bar, which would pile
|
|
//--- several sample rows onto the same clamped label and manufacture association out of nothing.
|
|
//--- THE PAD IS FIXED, NOT |labelBarOffset|. Two builds are only comparable row by row if they enumerate
|
|
//--- the same bars with the same stride, and both `lo` and `stride` below are derived from this range -
|
|
//--- so padding by the requested offset would move every row of the offset build. That is exactly what
|
|
//--- broke the positive control: it paired row k of an unshifted build with row k of a build starting
|
|
//--- `offset` bars later, whose label was then shifted a further `offset`, giving a pair 2*offset apart.
|
|
//--- The measured consequence was a control that reported the MI of labels 48 bars apart while claiming
|
|
//--- 24, failed its 5x gate, and voided every MI figure the EA printed.
|
|
int shiftPad = MiShiftPad();
|
|
if(MathAbs(labelBarOffset) > shiftPad || MathAbs(featureBarOffset) > shiftPad)
|
|
return -1; // caller asked for a shift the pad does not cover
|
|
lo += shiftPad;
|
|
hi -= shiftPad;
|
|
if(hi - lo < MI_MIN_SAMPLES)
|
|
return -1;
|
|
int stride = (int)MathMax(1, (hi - lo) / MI_SAMPLE_BARS);
|
|
//--- Published so the positive control can say how many BARS apart two sample rows are without
|
|
//--- recomputing this arithmetic at the call site, where it would silently drift out of agreement.
|
|
m_miStrideBars = stride;
|
|
int cap = (hi - lo) / stride + 1;
|
|
ArrayResize(cols, cap * m_neuronsCount);
|
|
ArrayResize(labels, cap);
|
|
if(continuousTarget)
|
|
ArrayResize(raw, cap);
|
|
int n = 0;
|
|
for(int i = lo; i < hi && n < cap; i += stride)
|
|
{
|
|
//--- Features come from bar i; the LABEL may be taken from a neighbouring bar (labelBarOffset != 0)
|
|
//--- so the caller can scan for a feature/label misalignment - see the alignment scan in
|
|
//--- ReportFeatureLabelInformation(). Both bars must carry a valid label for the row to count.
|
|
int li = i + labelBarOffset;
|
|
if(i >= ArraySize(m_labelCacheHasValue) || !m_labelCacheHasValue[i])
|
|
continue;
|
|
//--- The geometry scan asks "what WOULD this label be under a different barrier?", which by
|
|
//--- definition is not in the cache. Compute it on the spot instead - the cache belongs to the
|
|
//--- configured geometry and a scan must never write to it.
|
|
if(!m_barrierScanLiveLabels && (li < 0 || li >= ArraySize(m_labelCacheHasValue) || !m_labelCacheHasValue[li]))
|
|
continue;
|
|
if(m_barrierScanLiveLabels && (li < MathMax(m_barrierHorizonBars, 1) || li >= bars))
|
|
continue;
|
|
//--- BufferTempData(), NOT BufferTempDataCompute(). The Compute variant APPENDS the bar's features
|
|
//--- to TempData and never touches m_featureCache - only the caching wrapper writes that array. The
|
|
//--- first version of this function called Compute and then read m_featureCache, which
|
|
//--- ReInitADIndicators had just invalidated, so every column read back constant, FeatureColumnMI
|
|
//--- returned 0 for all of them, and all 17 candidates scored exactly 0.0000 nats. The tuner ran for
|
|
//--- 139 s per chart and always reported "no improvement" - a silent no-op that looked like a
|
|
//--- measurement. Read the values back out of TempData, which is where they actually land.
|
|
//--- FEATURE-side shift, distinct from labelBarOffset and not interchangeable with it. Shifting the
|
|
//--- LABEL changes which trade is being predicted, so at any non-zero offset the features sit INSIDE
|
|
//--- the labelled window and the score is lookahead - which is exactly what the alignment scan
|
|
//--- measures and correctly reports (4.7x more knowable 5 bars into a 128-bar window). Shifting the
|
|
//--- FEATURES instead keeps the label pinned to the entry bar and asks the honest question: does the
|
|
//--- state k bars BEFORE the entry still carry information about that entry's outcome? Positive k is
|
|
//--- strictly older (higher series index), so every row stays causal.
|
|
TempData.Clear();
|
|
if(!BufferTempData(i + featureBarOffset) || TempData.Total() < m_neuronsCount)
|
|
continue;
|
|
for(int f = 0; f < m_neuronsCount; f++)
|
|
cols[n * m_neuronsCount + f] = TempData.At(f);
|
|
if(continuousTarget)
|
|
{
|
|
//--- Excursions come from the cache only. The geometry scan's live-relabel path deliberately
|
|
//--- does not feed them: excursions do not depend on SL/TP at all (see the accumulators in
|
|
//--- TripleBarrierLabel), so re-deriving them per candidate geometry would compute the same
|
|
//--- number repeatedly and invite the impression that it varies with the barrier.
|
|
if(li >= ArraySize(m_excUpCache))
|
|
continue;
|
|
double up = m_excUpCache[li];
|
|
double dn = m_excDownCache[li];
|
|
if(!MathIsValidNumber(up) || !MathIsValidNumber(dn))
|
|
continue;
|
|
//--- A bar that TripleBarrierLabel() could not resolve (no valid ATR or close, typically the
|
|
//--- oldest bars) is still flagged as having a label, but its excursions were cleared to zero
|
|
//--- rather than measured. Price cannot genuinely travel zero in BOTH directions over a whole
|
|
//--- horizon, so this is an unambiguous "not measured" marker. Dropping those rows matters more
|
|
//--- than it looks: under EQUAL-FREQUENCY binning a block of identical zeros drags the lowest
|
|
//--- cut point onto zero, and a third of the sample then lands in one bin carrying no
|
|
//--- information - which would show up as a depressed score and read as "not predictable".
|
|
if(up <= 0.0 && dn <= 0.0)
|
|
continue;
|
|
if(target == MI_TARGET_EXC_UP)
|
|
raw[n] = up;
|
|
else
|
|
if(target == MI_TARGET_EXC_DOWN)
|
|
raw[n] = dn;
|
|
else
|
|
if(target == MI_TARGET_EXC_RANGE)
|
|
raw[n] = up + dn;
|
|
else
|
|
if(target == MI_TARGET_EXC_ASYM)
|
|
raw[n] = up - dn;
|
|
else
|
|
{
|
|
//--- Scale-free asymmetry. The denominator is > 0 here because rows with both
|
|
//--- excursions zero were dropped above, so no guard is needed beyond that.
|
|
raw[n] = (up - dn) / (up + dn); // MI_TARGET_EXC_ASYM_NORM
|
|
}
|
|
labels[n] = 0; // assigned below, once the distribution is known
|
|
}
|
|
else
|
|
if(m_barrierScanLiveLabels)
|
|
{
|
|
ENUM_SIGNAL v = TripleBarrierLabel(li);
|
|
if(v == Neutral && m_lastBarrierTimedOut)
|
|
m_barrierScanTimeouts++;
|
|
labels[n] = (v == Buy) ? 0 : ((v == Sell) ? 1 : 2);
|
|
}
|
|
else
|
|
labels[n] = m_labelCacheBuy[li] ? 0 : (m_labelCacheSell[li] ? 1 : 2);
|
|
n++;
|
|
}
|
|
TempData.Clear();
|
|
//--- EQUAL-FREQUENCY DISCRETISATION into the same 3 classes FeatureColumnMI's joint table expects, so
|
|
//--- every downstream piece - the block permutation, the null, the p-value, the lag profile - works on
|
|
//--- a continuous target with no change at all. Equal-frequency rather than equal-width because these
|
|
//--- distributions are fat-tailed (MFE especially): fixed-width bins would put almost every row in the
|
|
//--- first bin and measure nothing. It also fixes H(Y) at ln(3) = 1.099 nats for all four excursion
|
|
//--- targets, which makes their scores directly comparable to each other AND to the barrier label's
|
|
//--- ~1.02 - a comparison that would otherwise be confounded by class balance.
|
|
if(continuousTarget && n > 0)
|
|
{
|
|
double sorted[];
|
|
ArrayResize(sorted, n);
|
|
ArrayCopy(sorted, raw, 0, 0, n);
|
|
ArraySort(sorted);
|
|
double cut1 = sorted[n / 3];
|
|
double cut2 = sorted[(2 * n) / 3];
|
|
//--- A degenerate target (every value identical, e.g. a cache that never filled) would land every
|
|
//--- row in one class and score a flat zero. Say so rather than reporting the zero as a finding.
|
|
if(cut1 == cut2 && sorted[0] == sorted[n - 1])
|
|
{
|
|
Print(ID + ": MI excursion target " + IntegerToString(target) + " is CONSTANT across all "
|
|
+ IntegerToString(n) + " sampled bars - the excursion cache did not fill. Treating as "
|
|
"unusable rather than reporting its zero score as a measurement.");
|
|
return -1;
|
|
}
|
|
for(int q = 0; q < n; q++)
|
|
labels[q] = (raw[q] <= cut1) ? 0 : ((raw[q] <= cut2) ? 1 : 2);
|
|
}
|
|
return n;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Score an already-extracted sample. Split out from the extraction |
|
|
//| above so the permutation test can reuse ONE sample across every |
|
|
//| draw: feature extraction dominates the cost, and re-running it |
|
|
//| per shuffle is what would have made a few hundred permutations |
|
|
//| unaffordable. The shuffle is in place and destructive, which is |
|
|
//| harmless - composing permutations still yields a uniform |
|
|
//| permutation, so successive draws stay independent - but it does |
|
|
//| mean the OBSERVED (unshuffled) statistic must be taken first. |
|
|
//+------------------------------------------------------------------+
|
|
double CExpertSignalAIBase::ScoreMiSample(const double &cols[], int &labels[], int n, bool shuffleLabels)
|
|
{
|
|
if(n < MI_MIN_SAMPLES)
|
|
return -1.0;
|
|
//--- PERMUTATION BASELINE. Mutual information estimated from finite samples is biased UPWARD - with
|
|
//--- MI_BINS bins and 3 classes the bias is roughly (bins-1)(classes-1)/(2n) nats, which at these
|
|
//--- sample sizes is the same order as any real edge in this domain. So a raw MI figure is
|
|
//--- uninterpretable on its own: 0.004 nats could be a genuine weak signal or could be pure noise.
|
|
//--- Shuffling the labels destroys every real association while leaving the sample size, the binning
|
|
//--- and the class proportions untouched, so the score it produces IS this dataset's noise floor,
|
|
//--- measured rather than approximated. Reporting the two together turns "0.0042 nats" into either
|
|
//--- "0.0042 against a 0.0041 floor" (nothing) or "0.0042 against a 0.0009 floor" (something).
|
|
//--- BLOCK permutation, not a free one, and the difference is the whole validity of the test.
|
|
//--- Triple-barrier labels OVERLAP: two sample rows less than m_barrierHorizonBars apart share most of
|
|
//--- their outcome window, so their labels are strongly dependent. A free Fisher-Yates shuffle destroys
|
|
//--- that dependence as well as the feature/label association, which makes the null distribution far
|
|
//--- NARROWER than the truth and hands out significance that isn't there. The 2026-08-01 symbol sweep
|
|
//--- showed it in the raw: excess tracked the sampling STRIDE almost monotonically, and the three D1
|
|
//--- cells - where the stride had collapsed to 1-5 bars against a 128-bar horizon, i.e. ~99% window
|
|
//--- overlap - returned 5-9x the "signal" of every H1 cell at p=0.005. That was label autocorrelation
|
|
//--- leaking through an independence assumption, not an edge. It is Lopez de Prado ch. 4's non-IID
|
|
//--- problem arriving through the back door of the significance test.
|
|
//--- Permuting whole CONTIGUOUS BLOCKS at least one horizon long preserves the autocorrelation inside a
|
|
//--- block while destroying any feature/label association across blocks - so the null keeps the
|
|
//--- dependence structure and the p-value means what it says. It also degrades honestly: when overlap is
|
|
//--- severe there are few blocks, the null is correspondingly wide, and nothing reaches significance,
|
|
//--- which is the correct answer rather than a flattering one.
|
|
if(shuffleLabels)
|
|
{
|
|
int blockRows = (m_miStrideBars > 0)
|
|
? (int)MathCeil((double)MathMax(m_barrierHorizonBars, 1) / m_miStrideBars) : 1;
|
|
if(blockRows < 1)
|
|
blockRows = 1;
|
|
if(blockRows > n)
|
|
blockRows = n;
|
|
int blocks = (n + blockRows - 1) / blockRows;
|
|
m_miNullBlocks = blocks;
|
|
//--- Fisher-Yates over BLOCK ORDER; within-block order is left untouched, which is what preserves
|
|
//--- the local dependence. Copied out rather than swapped in place because blocks are not
|
|
//--- interchangeable in size - the last one is short whenever blockRows does not divide n.
|
|
int order[];
|
|
ArrayResize(order, blocks);
|
|
for(int b = 0; b < blocks; b++)
|
|
order[b] = b;
|
|
for(int b = blocks - 1; b > 0; b--)
|
|
{
|
|
//--- ShuffleRandomIndex, not MathRand()%: with blockRows == 1 the block count equals the row
|
|
//--- count, which can exceed MathRand()'s 15-bit range - same bias as the pass-2 queue shuffle.
|
|
int j = ShuffleRandomIndex(b + 1);
|
|
int t = order[b];
|
|
order[b] = order[j];
|
|
order[j] = t;
|
|
}
|
|
int shuffled[];
|
|
ArrayResize(shuffled, n);
|
|
int w = 0;
|
|
for(int b = 0; b < blocks && w < n; b++)
|
|
{
|
|
int src = order[b] * blockRows;
|
|
for(int q = 0; q < blockRows && w < n; q++)
|
|
{
|
|
int s = src + q;
|
|
shuffled[w++] = (s < n) ? labels[s] : labels[n - 1];
|
|
}
|
|
}
|
|
for(int i = 0; i < n; i++)
|
|
labels[i] = shuffled[i];
|
|
}
|
|
//--- H(Y) over the sampled labels, so the caller can express MI as a fraction of the information the
|
|
//--- label actually contains. Computed AFTER any shuffle, which leaves it unchanged by construction
|
|
//--- (a permutation preserves the class counts) - that invariance is itself a check on the shuffle.
|
|
int classCount[3] = {0, 0, 0};
|
|
for(int k = 0; k < n; k++)
|
|
classCount[labels[k]]++;
|
|
m_miLabelEntropy = 0.0;
|
|
for(int c = 0; c < 3; c++)
|
|
{
|
|
if(classCount[c] <= 0)
|
|
continue;
|
|
double pc = (double)classCount[c] / n;
|
|
m_miLabelEntropy -= pc * MathLog(pc);
|
|
}
|
|
double colVals[];
|
|
ArrayResize(colVals, n);
|
|
double total = 0.0;
|
|
m_miBestColumn = 0.0;
|
|
for(int f = 0; f < m_neuronsCount; f++)
|
|
{
|
|
for(int k = 0; k < n; k++)
|
|
colVals[k] = cols[k * m_neuronsCount + f];
|
|
double mi = FeatureColumnMI(colVals, labels, n);
|
|
total += mi;
|
|
if(mi > m_miBestColumn)
|
|
m_miBestColumn = mi;
|
|
}
|
|
return total / m_neuronsCount;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Extract + score in one call - the form the coordinate sweep uses, |
|
|
//| where each candidate genuinely needs a fresh extraction because |
|
|
//| the indicator settings (and therefore the features) just changed. |
|
|
//+------------------------------------------------------------------+
|
|
double CExpertSignalAIBase::ScoreCurrentParamsByMI(bool shuffleLabels = false)
|
|
{
|
|
double cols[];
|
|
int labels[];
|
|
//--- MI_TUNE_TARGET, not the barrier label - see the define's comment: the tuner selects
|
|
//--- indicator settings for the channel with measured signal (realised RANGE), not the one
|
|
//--- measured at the noise floor (direction).
|
|
int n = BuildMiSample(cols, labels, 0, 0, MI_TUNE_TARGET);
|
|
if(n < MI_MIN_SAMPLES)
|
|
return -1.0;
|
|
return ScoreMiSample(cols, labels, n, shuffleLabels);
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| FILTER-BASED indicator tuning. Replaced the genetic + successive- |
|
|
//| halving search on 2026-08-01. |
|
|
//| |
|
|
//| WHY THE GA HAD TO GO - measured, not assumed. Its cost was |
|
|
//| population x generations x rungs x seeds x eras-per-rung: |
|
|
//| rung 0: 8 cand x 3 seeds x 3 eras = 72 eras |
|
|
//| rung 1: 4 cand x 3 seeds x 8 eras = 96 |
|
|
//| rung 2: 2 cand x 3 seeds x 20 eras = 120 |
|
|
//| = 288 eras per generation x 4 generations = 1152 eras |
|
|
//| BEFORE the winner's real training started. Measured on SP500 H1: |
|
|
//| 9.3 h for the perceptron, 13.2 h for conv, ~48 h for LSTM and |
|
|
//| hybrid. Two days to tune is not a first-run experience. |
|
|
//| |
|
|
//| And it bought nothing. The space here is 90 points (10 MA periods |
|
|
//| x 9 MA types), so 1152 evaluations revisited each point ~13 times; |
|
|
//| meanwhile rungs of 3 and 8 eras cannot separate two MA periods at |
|
|
//| all - the 2026-08-01 run's finalists all scored 25.0-25.9% |
|
|
//| balanced accuracy, i.e. indistinguishable noise, and it then |
|
|
//| deployed the "winner" of that. |
|
|
//| |
|
|
//| THE REAL ERROR was using a full training run as the scoring |
|
|
//| function for a feature's period. The reference book does not: ch. |
|
|
//| 3.3 selects inputs by measuring each candidate indicator's |
|
|
//| CORRELATION with the target and dropping the ones with none, with |
|
|
//| no network involved. Mutual information is the same idea without |
|
|
//| the linearity assumption, which matters here because the label is |
|
|
//| 3-class categorical and the features are not monotonically related |
|
|
//| to it. Scoring is then arithmetic over cached features: seconds, |
|
|
//| not hours, and it scales with the number of enabled features |
|
|
//| rather than with topology cost - so LSTM tunes as fast as the MLP. |
|
|
//| |
|
|
//| COORDINATE SWEEP, not a product sweep: each parameter is optimised |
|
|
//| against the others' current values, one at a time. Cost is the SUM |
|
|
//| of the per-parameter candidate counts, not their product, so |
|
|
//| enabling every indicator stays affordable. Two passes, because the |
|
|
//| second can exploit what the first learned about the others; it |
|
|
//| stops early the moment a pass changes nothing. |
|
|
//| |
|
|
//| HONEST LIMIT, stated because it is the price of the trade: MI is a |
|
|
//| MARGINAL measure. It scores each feature column on its own, so a |
|
|
//| parameter that only pays off in combination with another can be |
|
|
//| missed. That is the standard filter-vs-wrapper tradeoff (Guyon & |
|
|
//| Elisseeff 2003). Given the wrapper here was ranking pure noise at |
|
|
//| 48 h a run, a fast marginal score is strictly the better deal. |
|
|
//| |
|
|
//| OBJECTIVE RE-POINTED 2026-08-16: candidates are scored against the |
|
|
//| excursion RANGE target (MI_TUNE_TARGET), not the barrier label - |
|
|
//| see the define's comment in ExpertSignalAIBase.mqh. Direction is |
|
|
//| measured noise, so the old objective gave the sweep a flat |
|
|
//| landscape and the gate rightly rejected every "winner"; RANGE is |
|
|
//| the channel with measured signal and the one the excursion head |
|
|
//| actually consumes these features for. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::TuneIndicatorsByFilter(void)
|
|
{
|
|
double best[];
|
|
m_indicatorTuner.Flatten(best);
|
|
double bestScore = ScoreCurrentParamsByMI();
|
|
if(bestScore < 0.0)
|
|
{
|
|
Print(ID + ": auto-tune skipped - not enough labelled in-sample bars to score indicator settings");
|
|
return;
|
|
}
|
|
double startScore = bestScore;
|
|
int evaluated = 0;
|
|
uint t0 = GetTickCount();
|
|
//--- SPREAD OF THE CANDIDATE SCORES. Without it "no improvement" is ambiguous between two readings
|
|
//--- that want opposite responses: INERT (trial scores identical to the incumbent because the
|
|
//--- parameter change never reaches the scored features, so `sc > bestScore` can never fire) versus
|
|
//--- LIVE and genuinely finding nothing. A spread of exactly zero says the first; a spread near the
|
|
//--- estimator's own noise says the second - and then the winner needs the family-wise gate the
|
|
//--- geometry scan and lag profile now carry, because installing a winner CHANGES THE FEATURE VECTOR
|
|
//--- and forces a fresh topology, a far heavier consequence than a printed row.
|
|
//---
|
|
//--- This measures the distinction directly, which is the point: the run-to-run evidence cannot settle
|
|
//--- it. "No improvement" on four consecutive runs (2026-08-05/06, 17 candidates) looks damning if the
|
|
//--- runs are treated as independent trials, but they are NOT - the scorer is deterministic and the
|
|
//--- runs cover nearly the same bars, so an incumbent that is the maximum on this data is the maximum
|
|
//--- on every run. That is one ~1-in-18 observation with three correlated repeats, not four of them.
|
|
//--- Note also that the INERT failure has already happened once here in a different form and was
|
|
//--- fixed (see the BufferTempData note in BuildMiSample: every candidate scored exactly 0.0000).
|
|
//--- Non-zero scores now mean that particular fault is gone.
|
|
double candMin = DBL_MAX, candMax = -DBL_MAX;
|
|
int readyMin = INT_MAX;
|
|
//--- The configured settings, kept so a winner that fails the gate below can be handed back. best[] is
|
|
//--- mutated in place by the descent, so it cannot serve as the restore point.
|
|
double configured[];
|
|
ArrayCopy(configured, best);
|
|
for(int pass = 0; pass < MI_TUNE_PASSES; pass++)
|
|
{
|
|
bool improvedThisPass = false;
|
|
for(int p = 0; p < AD_TUNE_PARAM_COUNT; p++)
|
|
{
|
|
//--- skip parameters whose indicator is switched off - they cannot affect the feature vector
|
|
int owner = m_indicatorTuner.ParamOwner(p);
|
|
bool on = (owner == 0 && m_useADCumulativeDelta) || (owner == 1 && m_useADShorteningOfThrust) ||
|
|
(owner == 2 && m_useADWyckoffEventStream) || (owner == 3 && m_useADWyckoffFailedStructure) ||
|
|
(owner == 4 && m_useADWyckoffSignificantBarInversion) || (owner == 5 && m_useMA) ||
|
|
(owner == 6 && m_useRSI) || (owner == 7 && m_useMACD) || (owner == 8 && m_useIchimoku);
|
|
if(!on)
|
|
continue;
|
|
double cands[];
|
|
int nc = m_indicatorTuner.ParamCandidates(p, cands);
|
|
double keep = best[p];
|
|
for(int c = 0; c < nc; c++)
|
|
{
|
|
//--- The longest uninterruptible stretch in the EA: every candidate re-creates handles,
|
|
//--- refreshes, and scores a full MI sample. Asked per candidate so a stop request costs at
|
|
//--- most one candidate rather than the rest of the descent - see ShutdownRequested().
|
|
if(ShutdownRequested())
|
|
{
|
|
//--- Hand the OPERATOR's settings back before leaving. best[] is mutated in place by the
|
|
//--- descent and the tuner object currently carries the LAST TRIAL's parameters, which
|
|
//--- nothing chose and which the .cfg would otherwise persist as if it had been selected.
|
|
//--- Deliberately no ReInitADIndicators/RefreshData here - those build indicator handles
|
|
//--- for a program that is being torn down, and spending the teardown budget on them is
|
|
//--- exactly what strands the chart objects.
|
|
m_indicatorTuner.Unflatten(configured);
|
|
PrintFormat("%s: auto-tune ABANDONED after %d candidates - stop requested. Configured"
|
|
" indicator settings restored; nothing installed.", ID, evaluated);
|
|
return;
|
|
}
|
|
if(cands[c] == keep)
|
|
continue; // already scored as the incumbent
|
|
double trial[];
|
|
ArrayCopy(trial, best);
|
|
trial[p] = cands[c];
|
|
m_indicatorTuner.Unflatten(trial);
|
|
ReInitADIndicators(m_indicatorsPtr); // also invalidates the feature cache (params changed)
|
|
//--- REFRESH, or the re-init changes nothing that the scorer can see. ReInitADIndicators
|
|
//--- creates a NEW handle carrying the new parameters and flags the feature cache stale, so
|
|
//--- features are genuinely recomputed - but BufferTempDataCompute() reads the CIndicatorBuffer
|
|
//--- objects, and only Refresh() copies data out of a handle into those. Without this the
|
|
//--- buffers still hold values copied from the PREVIOUS handle, so every candidate is scored on
|
|
//--- identical features. Measured on SP500 H1 2026-08-07: all 17 candidates returned exactly
|
|
//--- 0.00359 nats, a candidate-score span of 0.00000.
|
|
RefreshData();
|
|
int ready = TunableBarsCalculated();
|
|
if(ready >= 0)
|
|
readyMin = (int)MathMin(readyMin, ready);
|
|
double sc = ScoreCurrentParamsByMI();
|
|
evaluated++;
|
|
if(sc >= 0.0)
|
|
{
|
|
candMin = MathMin(candMin, sc);
|
|
candMax = MathMax(candMax, sc);
|
|
}
|
|
if(sc > bestScore)
|
|
{
|
|
bestScore = sc;
|
|
keep = cands[c];
|
|
improvedThisPass = true;
|
|
}
|
|
}
|
|
best[p] = keep;
|
|
}
|
|
if(!improvedThisPass)
|
|
break; // coordinate descent has converged - further passes cannot move anything
|
|
}
|
|
//--- SELECTION GATE. bestScore is a MAXIMUM over every candidate scored, so it carries the same defect
|
|
//--- the barrier-geometry winner test and the lag profile were fixed for: the maximum of N draws from a
|
|
//--- null sits well above any single draw, and installing on "it beat the incumbent" alone crowns noise.
|
|
//--- The stakes here are higher than either of those, because this one ACTS - it replaces the user's
|
|
//--- deliberate indicator settings and forces BuildFreshTopology(), so the network then trains on
|
|
//--- whatever the noise picked.
|
|
//---
|
|
//--- Test: draw the winner's own permutation null once (the sample is extracted once and every draw
|
|
//--- reshuffles it - see ScoreMiSample), take the per-candidate p, then correct it for having CHOSEN
|
|
//--- this candidate out of N with Sidak: p_family = 1 - (1 - p)^N. Sidak rather than an explicit
|
|
//--- max-of-N resample because each candidate here has a DIFFERENT feature set, so their draws cannot
|
|
//--- be pooled the way the geometry scan's can; Sidak needs only the one null and is exact under
|
|
//--- independence, mildly anti-conservative under positive dependence - stated rather than hidden.
|
|
//---
|
|
//--- WHAT THIS DOES NOT ESTABLISH: that the winner beats the INCUMBENT by a significant margin. It
|
|
//--- bounds the "best of N noise draws" failure, which is the one that was actually live here. Requiring
|
|
//--- bestScore > startScore as well means a change needs both an improvement and a defensible signal.
|
|
bool install = (bestScore > startScore);
|
|
double pFamily = 1.0;
|
|
int distinct = (int)MathMax(evaluated + 1, 1); // candidates scored, plus the incumbent
|
|
if(install)
|
|
{
|
|
double wc[];
|
|
int wl[];
|
|
//--- same target as the sweep's scorer, or the gate would test the winner against a
|
|
//--- different question than the one it was selected on
|
|
int wn = BuildMiSample(wc, wl, 0, 0, MI_TUNE_TARGET);
|
|
if(wn >= MI_MIN_SAMPLES)
|
|
{
|
|
double obs = ScoreMiSample(wc, wl, wn, false);
|
|
int atLeast = 0, draws = 0;
|
|
for(int s = 0; s < MI_NOISE_PERMUTATIONS; s++)
|
|
{
|
|
//--- A truncated null is not a smaller null, it is a WRONG one - fewer draws shifts p toward
|
|
//--- significance. So a stop here abandons the test entirely (draws stays 0, pFamily stays
|
|
//--- 1.0, install becomes false) rather than installing on a partial null.
|
|
if(ShutdownRequested())
|
|
{
|
|
draws = 0;
|
|
break;
|
|
}
|
|
double d = ScoreMiSample(wc, wl, wn, true);
|
|
if(d < 0.0)
|
|
continue;
|
|
if(d >= obs)
|
|
atLeast++;
|
|
draws++;
|
|
}
|
|
if(draws > 0)
|
|
{
|
|
double pSingle = (double)(1 + atLeast) / (draws + 1);
|
|
pFamily = 1.0 - MathPow(1.0 - pSingle, (double)distinct);
|
|
}
|
|
}
|
|
install = (pFamily <= MI_TUNE_ALPHA);
|
|
}
|
|
if(!install)
|
|
{
|
|
ArrayCopy(best, configured);
|
|
bestScore = startScore;
|
|
}
|
|
//--- install the winner and leave the indicators/feature cache consistent with it
|
|
m_indicatorTuner.Unflatten(best);
|
|
ReInitADIndicators(m_indicatorsPtr);
|
|
RefreshData();
|
|
//--- A gated INSTALL is chart-level news, not just this model's: persist the winning periods so
|
|
//--- the classic votes, the signal-DB key and every later tuner seed adopt them on the next
|
|
//--- attach (restart-grained - see Variables\TunedPeriods.mqh for why not mid-run).
|
|
if(install)
|
|
SaveTunedPeriods(m_indicatorTuner.maPeriod, m_indicatorTuner.maType, m_indicatorTuner.rsiPeriod,
|
|
m_indicatorTuner.macdFast, m_indicatorTuner.macdSlow, m_indicatorTuner.macdSignal,
|
|
m_indicatorTuner.ichiTenkan, m_indicatorTuner.ichiKijun, m_indicatorTuner.ichiSenkou);
|
|
double candSpread = (evaluated > 0 && candMax >= candMin) ? (candMax - candMin) : 0.0;
|
|
Print(ID + StringFormat(": auto-tune complete - %d candidate settings scored in %.1fs, "
|
|
"feature/label mutual information %.5f -> %.5f nats%s | candidate scores span "
|
|
"%.5f (%.5f..%.5f)%s",
|
|
evaluated, (GetTickCount() - t0) / 1000.0, startScore, bestScore,
|
|
(bestScore <= startScore ? " (no improvement - keeping the configured settings)" : ""),
|
|
candSpread, (evaluated > 0 ? candMin : 0.0), (evaluated > 0 ? candMax : 0.0),
|
|
(evaluated > 0 && candSpread <= 0.0
|
|
? StringFormat(" <-- ZERO SPREAD: every candidate scored identically, so the "
|
|
"parameter change is STILL not reaching the scored features even "
|
|
"with the post-re-init RefreshData(). Least-ready tunable handle "
|
|
"had %d bars calculated - if that is 0 or far below the study "
|
|
"window, the handles are simply not done calculating yet and the "
|
|
"tuner needs to yield between candidates rather than score them "
|
|
"back to back.", (readyMin == INT_MAX ? -1 : readyMin))
|
|
: StringFormat(" | winner %s (selection p=%.4f after correcting for %d "
|
|
"candidates, need <=%.2f)",
|
|
(install ? "INSTALLED" : "REJECTED - keeping the configured "
|
|
"settings, since the best of N noise draws beats its incumbent "
|
|
"almost every time"),
|
|
pFamily, distinct, MI_TUNE_ALPHA))));
|
|
//--- An EXACTLY zero score is not a weak feature set, it is a broken measurement. Mutual information
|
|
//--- estimated from finite samples is biased UPWARD - roughly (bins-1)(classes-1)/(2N) nats, ~0.0035
|
|
//--- here - so even columns of pure noise score above zero. Landing on 0.0000 means every column read
|
|
//--- back constant, which is what a feature-extraction fault looks like. Said out loud because the
|
|
//--- first version of this function did exactly that and reported it as "no improvement".
|
|
if(bestScore <= 0.0)
|
|
Print(ID + ": WARNING - every candidate scored 0.0000 nats. Finite-sample bias alone should put "
|
|
"noise above zero, so this indicates the feature values are not being read, not that the "
|
|
"features are uninformative. Indicator settings left at their configured values.");
|
|
ReportFeatureLabelInformation();
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| "Do these features predict this label at all?" - answered without |
|
|
//| training, topology or convergence, so unlike every accuracy |
|
|
//| number in this codebase it cannot be confounded by an optimizer |
|
|
//| or an objective. |
|
|
//| |
|
|
//| DELIBERATELY SEPARATE FROM THE TUNER, and not gated on era 0 with |
|
|
//| it. The sweep must only run on a fresh model - re-tuning would |
|
|
//| change the input vector out from under weights already fitted to |
|
|
//| the old one - but this reads the same cached features and writes |
|
|
//| nothing, so tying it to that gate meant the only way to see the |
|
|
//| answer was to bin a model mid-run (45 trained eras, on 2026-08-01) |
|
|
//| purely to re-ask a read-only question. Runs once per attach. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ReportFeatureLabelInformation(void)
|
|
{
|
|
m_miReportDone = true;
|
|
//--- PERMUTATION TEST, done properly. Build the current settings' sample ONCE, take the observed
|
|
//--- statistics from it, then reuse that same sample for every null draw - extraction is the expensive
|
|
//--- part, so this makes a few hundred permutations cost about what five used to.
|
|
//---
|
|
//--- Five was not enough, and the 2026-08-01 log is the proof: all four charts scored the IDENTICAL
|
|
//--- 0.00401 nats on identical features and identical labels, yet reported z of +1.3, +2.0, +4.0 and
|
|
//--- +4.7 - two "at the noise floor", two "real". The whole swing came from estimating the null's spread
|
|
//--- from five draws, where the standard deviation of the standard-deviation estimate is ~35%. The
|
|
//--- denominator was noisier than the effect.
|
|
//---
|
|
//--- So: no z-score and no normality assumption. An EMPIRICAL p-value, counting how many null draws
|
|
//--- reached the observed value, with the +1/(B+1) correction (Phipson & Smyth 2010) that keeps p from
|
|
//--- ever being reported as exactly zero - the test can only ever bound p below by 1/(B+1).
|
|
double cols[];
|
|
int labels[];
|
|
int nSample = BuildMiSample(cols, labels);
|
|
double observed = (nSample >= MI_MIN_SAMPLES) ? ScoreMiSample(cols, labels, nSample, false) : -1.0;
|
|
double signalBestCol = m_miBestColumn;
|
|
double labelEntropy = m_miLabelEntropy;
|
|
double floorSum = 0.0, floorSumSq = 0.0, floorBestColSum = 0.0;
|
|
int draws = 0, atLeastMean = 0, atLeastBestCol = 0;
|
|
uint tPerm = GetTickCount();
|
|
for(int s = 0; observed >= 0.0 && s < MI_NOISE_PERMUTATIONS; s++)
|
|
{
|
|
//--- A few hundred full MI scorings, and nothing below can start until they finish. Abandon the
|
|
//--- test outright rather than truncate it: fewer draws does not make a smaller null, it makes a
|
|
//--- WRONG one (p shifts toward significance), and this null is what licenses the direction target.
|
|
if(ShutdownRequested())
|
|
{
|
|
draws = 0;
|
|
break;
|
|
}
|
|
double sc = ScoreMiSample(cols, labels, nSample, true);
|
|
if(sc < 0.0)
|
|
continue;
|
|
floorSum += sc;
|
|
floorSumSq += sc * sc;
|
|
floorBestColSum += m_miBestColumn;
|
|
if(sc >= observed)
|
|
atLeastMean++;
|
|
//--- The MAX over columns is compared against the null distribution OF THE MAX, which corrects for
|
|
//--- testing 26 features at once by construction - no Bonferroni needed, and far less conservative.
|
|
if(m_miBestColumn >= signalBestCol)
|
|
atLeastBestCol++;
|
|
draws++;
|
|
}
|
|
//--- Left the null early because the program is going away: everything from here on either prints a
|
|
//--- number derived from `draws` or starts another scan. Neither is worth a millisecond of the teardown
|
|
//--- budget, and m_dirEvidence staying false is the safe direction (a torn-down run deploys nothing).
|
|
if(ShutdownRequested())
|
|
return;
|
|
double floorMean = (draws > 0) ? floorSum / draws : -1.0;
|
|
double floorVar = (draws > 1) ? MathMax(0.0, floorSumSq / draws - floorMean * floorMean) : 0.0;
|
|
double floorSd = MathSqrt(floorVar * (draws > 1 ? (double)draws / (draws - 1) : 1.0));
|
|
double floorBestCol = (draws > 0) ? floorBestColSum / draws : -1.0;
|
|
double pMean = (draws > 0) ? (double)(1 + atLeastMean) / (draws + 1) : 1.0;
|
|
double pBestCol = (draws > 0) ? (double)(1 + atLeastBestCol) / (draws + 1) : 1.0;
|
|
//--- Two SEPARATE questions, because at these sample sizes a small p can accompany a worthless effect.
|
|
//--- (1) Is it real - the p-values. (2) Is it big enough to trade - the excess as a share of H(Y), i.e.
|
|
//--- of everything there is to know about the label. Both are printed; neither is collapsed into a verdict
|
|
//--- that hides the other.
|
|
double excessShare = (labelEntropy > 1e-9 && floorMean >= 0.0)
|
|
? 100.0 * (observed - floorMean) / labelEntropy : 0.0;
|
|
string verdict = (draws > 0 && pMean <= 0.05)
|
|
? "above the noise floor - a real association"
|
|
: "AT THE NOISE FLOOR - indistinguishable from shuffled labels";
|
|
//--- SCREEN, not commentary. This test asks directly whether the features carry information about the
|
|
//--- DIRECTIONAL barrier label, so clearing it licenses the direction target on its own - see
|
|
//--- m_dirEvidence. It is an OR with the excursion-asymmetry screen because the two look for the same
|
|
//--- thing by different routes and either finding is sufficient; requiring both would reject on the
|
|
//--- weaker of two independent measurements.
|
|
if(draws > 0 && pMean <= 0.05)
|
|
{
|
|
m_dirEvidence = true;
|
|
m_dirEvidenceWhy = "feature/label mutual information cleared its block-permuted null";
|
|
}
|
|
//--- Name the feature vector this was measured on. These numbers are only about the model if the two
|
|
//--- match, and on 2026-08-02 they did not: the report ran before the cross-asset panel existed and
|
|
//--- silently described a narrower vector than training used. Stating the width and the panel's
|
|
//--- presence makes that mismatch visible in the log instead of requiring a timestamp comparison.
|
|
string vecNote = StringFormat("%d features/bar, cross-asset %s", m_neuronsCount,
|
|
m_crossAsset.IsReady()
|
|
? "PRESENT"
|
|
: "ABSENT (reference symbols unsynchronised - these numbers describe "
|
|
"a NARROWER vector than training will use)");
|
|
Print(ID + StringFormat(": feature/label information - %.5f nats/feature vs a shuffled-label null of "
|
|
"%.5f +/- %.5f over %d permutations, p=%.4f; strongest single feature %.5f vs "
|
|
"%.5f (null max, p=%.4f); excess is %.2f%% of the label's %.3f nats of entropy "
|
|
"(%d samples %d bars apart = %d independent blocks over a %d-bar horizon, "
|
|
"%.1fs) [%s]. %s.",
|
|
observed, floorMean, floorSd, draws, pMean,
|
|
signalBestCol, floorBestCol, pBestCol, excessShare, labelEntropy,
|
|
nSample, m_miStrideBars, m_miNullBlocks, m_barrierHorizonBars,
|
|
(GetTickCount() - tPerm) / 1000.0, vecNote, verdict));
|
|
//--- POWER, stated up front. The block permutation above makes the p-value HONEST under overlapping
|
|
//--- labels, but it cannot manufacture information that overlap destroyed: when the sampling stride is
|
|
//--- far shorter than the horizon there are few genuinely independent blocks, and a handful of blocks
|
|
//--- cannot resolve an effect this small however many rows they contain. Saying so prevents the opposite
|
|
//--- error to the one this replaced - reading "not significant" as "no signal" when it means "not enough
|
|
//--- independent data to tell".
|
|
if(m_miNullBlocks > 0 && m_miNullBlocks < 30)
|
|
Print(ID + StringFormat(": NOTE - only %d independent label blocks in this sample (%d-bar horizon, "
|
|
"%d-bar sampling stride). The rows overlap heavily, so this test has little "
|
|
"power: treat a non-significant result here as 'not enough independent "
|
|
"history to answer', not as 'no signal'. More history, or a shorter horizon, "
|
|
"is what would settle it.", m_miNullBlocks, m_barrierHorizonBars,
|
|
m_miStrideBars));
|
|
//--- Stated every time, not only on a bad result: this measure is MARGINAL and PER-BAR, while the network
|
|
//--- reads m_historyBars bars at once. It can therefore only ever prove that signal EXISTS, never that it
|
|
//--- does not - an interaction across features or across time is invisible to it by construction. Said
|
|
//--- out loud so a floor-level reading is not over-read into "this instrument is unpredictable".
|
|
if(!(draws > 0 && pMean <= 0.05))
|
|
Print(ID + ": NOTE - that measure is marginal (one feature at a time) and per-bar, whereas the "
|
|
"network sees " + IntegerToString((int)m_historyBars) + " bars jointly. A floor-level reading "
|
|
"rules out a simple per-feature edge; it cannot rule out one that only exists in combination "
|
|
"or across time. It does mean no per-feature indicator retuning will help.");
|
|
if(observed < 0.0)
|
|
return;
|
|
//--- POSITIVE CONTROL. Three separate "measurements" in this codebase have turned out to be silent
|
|
//--- no-ops that produced plausible numbers (the MI scorer reading an array nobody filled; the
|
|
//--- eval-mode guard that switched off the imbalance correction; the alternation gate whose premise was
|
|
//--- never true). A floor reading is therefore worthless until the instrument is shown to respond to a
|
|
//--- signal that is KNOWN to be there. This one is free: the label of a NEIGHBOURING sample row. Rows are
|
|
//--- `stride` bars apart, far inside the barrier horizon, so their outcome windows overlap heavily and
|
|
//--- the two labels must be strongly associated. Fed through the identical binning and estimator as every
|
|
//--- other column. If THIS lands near the floor, the estimator is broken and no MI number above means
|
|
//--- anything; if it lands far above, a floor reading on the real features can be believed.
|
|
//--- The control pairs each row's label with the label of a bar a FIXED, KNOWN distance away, so the two
|
|
//--- outcome windows overlap heavily and must be strongly associated.
|
|
//--- THIS CONTROL HAS NOW CRIED WOLF TWICE, AND BOTH TIMES THE ESTIMATOR WAS INNOCENT.
|
|
//--- 2026-08-01 it paired with the NEXT SAMPLE ROW, whose distance is the sampling stride - and stride
|
|
//--- varies with how much history a symbol has, so the control's strength varied with the
|
|
//--- cell rather than with the estimator. All three M5 cells (stride 160-717 bars against a
|
|
//--- 128-bar horizon, i.e. windows that do not overlap AT ALL) voided their own results.
|
|
//--- 2026-08-02 the range was padded by |offset|, which moved the offset build's FIRST BAR as well as
|
|
//--- its label, so row k of one build sat `offset` bars from row k of the other and the
|
|
//--- label was shifted a further `offset`: the pair was 2x as far apart as reported. On
|
|
//--- SP500 H1 it printed 0.00307 nats for "24 bars apart" - which is the true value for 48
|
|
//--- bars - failed its 5x gate, and stamped "every mutual-information figure above is void"
|
|
//--- on measurements that were fine. Confirmed by computing the same quantity independently
|
|
//--- in research/test_mi_control.py: 0.01655 at 24 bars, 0.00298 at 48.
|
|
//--- The lesson both share: A CONTROL THAT DEPENDS ON THE THING IT CERTIFIES CANNOT CERTIFY IT. Pin the
|
|
//--- control's distance to something the data cannot move, and make it a distance where the association
|
|
//--- is overwhelming rather than marginal - hence the adjacent bar below.
|
|
//--- TWO distances, and the GATE is the adjacent bar. Its barrier window overlaps the reference one by
|
|
//--- (h-1)/h, so "these must be associated" is unarguable, and unlike a horizon-relative offset it does
|
|
//--- not vary with the horizon, the stride or the symbol. The quarter-horizon figure is kept as a
|
|
//--- DIAGNOSTIC because it says something the gate cannot: how fast a triple-barrier label decorrelates.
|
|
//--- Measured independently on SP500 H1 (research/test_mi_control.py, 74k bars): 0.542 nats at 1 bar,
|
|
//--- 0.017 at 24, 0.003 at 48, against a ~0.002 floor. Note what that means - a quarter-horizon control
|
|
//--- clears a 5x gate by under 2x even when everything is working, which is far too little headroom for
|
|
//--- the one measurement whose job is to certify all the others.
|
|
//--- The headline verdict is printed; everything from here down is corroboration (a positive control,
|
|
//--- then a nine-offset alignment scan) and each step is another full MI sample build. Diagnostics are
|
|
//--- worth their cost while the EA is running and worth nothing while it is being torn down.
|
|
if(ShutdownRequested())
|
|
return;
|
|
double controlMi = -1.0, decorrMi = -1.0;
|
|
int controlBars = 1;
|
|
int decorrBars = MathMax(1, MathMax(m_barrierHorizonBars, 1) / 4);
|
|
{
|
|
//--- Rebuilt rather than reused because the permutation loop above destroyed the honest label
|
|
//--- ordering, and controlling against a shuffled array would measure the floor twice.
|
|
double c0[], cK[];
|
|
int l0[], lK[];
|
|
int n0 = BuildMiSample(c0, l0);
|
|
if(n0 >= MI_MIN_SAMPLES)
|
|
{
|
|
int offs[2];
|
|
offs[0] = controlBars;
|
|
offs[1] = decorrBars;
|
|
for(int oi = 0; oi < 2; oi++)
|
|
{
|
|
int nK = BuildMiSample(cK, lK, offs[oi]);
|
|
//--- Both builds are padded by the SAME fixed amount, so they enumerate the same bars with
|
|
//--- the same stride and row k of one is row k of the other. Sized from what actually came
|
|
//--- back, never from the caller's count.
|
|
int nc = MathMin(n0, nK);
|
|
if(nc < MI_MIN_SAMPLES)
|
|
continue;
|
|
double neighbourLabel[];
|
|
int selfLabels[];
|
|
ArrayResize(neighbourLabel, nc);
|
|
ArrayResize(selfLabels, nc);
|
|
for(int k = 0; k < nc; k++)
|
|
{
|
|
selfLabels[k] = l0[k];
|
|
neighbourLabel[k] = (double)lK[k];
|
|
}
|
|
double v = FeatureColumnMI(neighbourLabel, selfLabels, nc);
|
|
if(oi == 0)
|
|
controlMi = v;
|
|
else
|
|
decorrMi = v;
|
|
}
|
|
}
|
|
}
|
|
Print(ID + StringFormat(": MI positive control - the ADJACENT bar's label (windows overlap %d of %d bars) "
|
|
"scores %.5f nats against the ~%.5f noise floor; by a quarter horizon (%d bars) "
|
|
"it is already down to %.5f, which is how fast this target decorrelates. %s",
|
|
MathMax(m_barrierHorizonBars, 1) - 1, MathMax(m_barrierHorizonBars, 1),
|
|
controlMi, floorMean, decorrBars, decorrMi,
|
|
(controlMi > floorMean * 5.0)
|
|
? "The estimator detects a known association on this exact data, so a "
|
|
"floor-level reading above is a real finding and not a broken measurement."
|
|
: "WARNING - the estimator FAILED to detect an association that must be there. "
|
|
"Every mutual-information figure above is void; fix this before drawing any "
|
|
"conclusion from them."));
|
|
//--- ALIGNMENT SCAN. A floor reading has two very different causes: the features genuinely do not predict
|
|
//--- this target, or they DO and something upstream has knocked the two out of step (an off-by-one in the
|
|
//--- label index, a horizon applied to the wrong bar, a feature window that lags what it claims). Both
|
|
//--- destroy the information before any topology sees it, and both look identical in every accuracy number
|
|
//--- this EA prints - which is exactly why four different architectures all landed on the same precision.
|
|
//--- Re-scoring against the label taken from bar i+k separates them: a peak at some k != 0 IS a
|
|
//--- misalignment (and names its size), a flat profile says the features simply do not carry this target.
|
|
//--- THE TWO DIRECTIONS ARE NOT SYMMETRIC, and the first version of this scan treated them as if they
|
|
//--- were - it read the k>0 rise as a misalignment and cried "fix this before concluding anything",
|
|
//--- which was a false alarm produced by the diagnostic's own design.
|
|
//---
|
|
//--- Bar indices here are MQL5 SERIES indices: HIGHER index = OLDER bar (TripleBarrierLabel walks its
|
|
//--- window with `for(t = idx-1; t >= idx-horizon; t--)`, i.e. decreasing index = forward in time).
|
|
//--- So:
|
|
//--- k < 0 the label belongs to a NEWER bar, whose barrier window opens AFTER the features exist.
|
|
//--- Nothing at bar i can legitimately know it. A peak here is real LOOKAHEAD and is a bug.
|
|
//--- k > 0 the label belongs to an OLDER bar, whose window is already k bars into its life by the
|
|
//--- time bar i happens - so the features at bar i legitimately contain the realised first k
|
|
//--- bars of that outcome. MI MUST rise with k. That is arithmetic, not a defect.
|
|
//--- Only the k<0 side can indict the pipeline. The k>0 side is a second positive control, and its
|
|
//--- GRADIENT is the useful number: it says how fast a barrier outcome becomes knowable once the window
|
|
//--- is running, against how little is knowable at entry (k=0).
|
|
int offsets[] = { -5, -3, -2, -1, 0, 1, 2, 3, 5 };
|
|
string profile = "";
|
|
double atZero = -1.0, worstFuture = -1.0, farPast = -1.0;
|
|
int worstFutureK = 0;
|
|
for(int oi = 0; oi < ArraySize(offsets); oi++)
|
|
{
|
|
//--- Nine sample builds. A partial profile cannot be read - the lookahead test compares the k<0
|
|
//--- side against k=0 and both must exist - so a stop abandons the scan rather than printing a row
|
|
//--- with holes in it that would look like a null result at the missing offsets.
|
|
if(ShutdownRequested())
|
|
return;
|
|
double oc[];
|
|
int ol[];
|
|
int on = BuildMiSample(oc, ol, offsets[oi]);
|
|
double os = (on >= MI_MIN_SAMPLES) ? ScoreMiSample(oc, ol, on, false) : -1.0;
|
|
profile += StringFormat("%s%+d:%.5f", (oi > 0 ? " " : ""), offsets[oi], os);
|
|
if(offsets[oi] == 0)
|
|
atZero = os;
|
|
else
|
|
if(offsets[oi] < 0 && os > worstFuture)
|
|
{
|
|
worstFuture = os;
|
|
worstFutureK = offsets[oi];
|
|
}
|
|
else
|
|
if(offsets[oi] > 0)
|
|
farPast = os; // offsets ascend, so this ends on the largest k
|
|
}
|
|
//--- A MARGIN, not a bare comparison. Every one of these offsets is an estimate with the same noise as
|
|
//--- the headline statistic, so "k=-3 came out above k=0" is meaningless when the gap is smaller than the
|
|
//--- null's own spread. Shipped without this, the 2026-08-01 sweep flagged LOOKAHEAD on 7 of 12 cells on
|
|
//--- gaps of 0.00008-0.00040 nats against a measured null sd of ~0.00030 - all noise, every one. Three
|
|
//--- SDs is the same discipline the deploy floor already applies to precision: an anomaly has to clear
|
|
//--- the measurement error before it gets a name. (Third time this session that comparing two point
|
|
//--- estimates without their spread produced a confident wrong answer - see MI_NOISE_PERMUTATIONS.)
|
|
double lookaheadMargin = 3.0 * floorSd;
|
|
string alignVerdict;
|
|
if(worstFuture > atZero + lookaheadMargin)
|
|
alignVerdict = StringFormat(" | LOOKAHEAD - k=%d (a label whose barrier window opens AFTER these "
|
|
"features exist) scores %.5f against %.5f at k=0, clearing the %.5f "
|
|
"margin (3 sd of the null). The features can only score there by "
|
|
"containing future information. Fix that before trusting any accuracy "
|
|
"number this EA prints.", worstFutureK, worstFuture, atZero, lookaheadMargin);
|
|
else
|
|
alignVerdict = StringFormat(" | clean: no future label (k<0) beats k=0, so there is no lookahead. "
|
|
"The rise on the k>0 side is expected - those windows are already open, "
|
|
"so the features hold part of the answer - and its size is the finding: "
|
|
"%.5f at k=+5 against %.5f at k=0, i.e. ~%.1fx more is knowable %d bars "
|
|
"into a %d-bar window than at the entry the model actually trades.",
|
|
farPast, atZero, (atZero > 1e-9 ? farPast / atZero : 0.0), 5,
|
|
m_barrierHorizonBars);
|
|
Print(ID + ": MI label-alignment scan (label from bar i+k; higher index = OLDER bar, so k<0 is the "
|
|
"future) - " + profile + alignVerdict);
|
|
ReportFeatureLagProfile();
|
|
//--- Runs after the lag profile and before the geometry scan on purpose: the geometry scan chooses
|
|
//--- among SL/TP pairings, and this asks whether predicting SL/TP is a well-posed problem at all.
|
|
//--- Reading them in that order stops a geometry winner from being interpreted as evidence that the
|
|
//--- exit is learnable.
|
|
ReportExcursionInformation();
|
|
ReportBarrierGeometryScan();
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| WHICH BARRIER GEOMETRY IS ACTUALLY PREDICTABLE AT ENTRY. |
|
|
//| |
|
|
//| The alignment scan established the shape of the problem: 4.7x more |
|
|
//| is knowable 5 bars into a 128-bar window than at the entry the |
|
|
//| model trades on. A 6xATR target reached over 128 bars is decided |
|
|
//| overwhelmingly by what happens DURING the window, so whatever the |
|
|
//| entry state knows is buried under 128 bars of subsequent noise. |
|
|
//| That is a property of the TARGET, and no topology can undo it - |
|
|
//| which is why four different architectures all landed on precision |
|
|
//| exactly equal to the base rate. |
|
|
//| |
|
|
//| So measure the target instead of guessing at it. For each SL/TP |
|
|
//| pairing the user can actually select, relabel the same sampled |
|
|
//| bars and score how much the SAME features say about THAT outcome. |
|
|
//| Seconds, no training, no topology. |
|
|
//| |
|
|
//| RANKED ON EXCESS OVER ITS OWN NULL, IN NATS. The first version |
|
|
//| divided that by the geometry's own H(Y), reasoning that each label |
|
|
//| has a different amount of information available to find. That was |
|
|
//| backwards and it produced a wrong answer on the first run: it |
|
|
//| named 3:10, whose horizon is CLAMPED (it wants ~320 bars and gets |
|
|
//| BARRIER_HORIZON_MAX), so most trades never resolve, Neutral |
|
|
//| dominates, H(Y) collapses - and dividing by a collapsing |
|
|
//| denominator made the most degenerate label look like the most |
|
|
//| predictable one. Subtracting each geometry's own measured null |
|
|
//| already removes the class-balance bias, which is the only thing |
|
|
//| the normalisation was needed for. |
|
|
//| |
|
|
//| A clamped geometry is DISQUALIFIED outright, not merely ranked |
|
|
//| down. The deployed EA holds until SL or TP with no bar limit, so a |
|
|
//| truncated label trains the model on a question the strategy never |
|
|
//| asks. Directional share is printed for the same reason: a label |
|
|
//| nobody can trade is not a candidate however well it scores. |
|
|
//| |
|
|
//| What it cannot tell you: chance precision equals the break-even |
|
|
//| win rate at every geometry (both are m/(m+k) under a driftless |
|
|
//| walk), so a tighter target does NOT buy expectancy on its own. It |
|
|
//| buys PREDICTABILITY - a shorter window has less noise piled on top |
|
|
//| of what the entry state knows. The ranking finds where the signal |
|
|
//| is largest; it is still on the model to convert it. |
|
|
//+------------------------------------------------------------------+
|
|
//+------------------------------------------------------------------+
|
|
//| HOW FAR BACK THE FEATURES STILL SAY ANYTHING - see the declaration.|
|
|
//| |
|
|
//| Returns the deepest lag whose score clears the null, or 0 when |
|
|
//| none does. Read-only; the caller decides what to do with it. |
|
|
//| |
|
|
//| The null is redrawn PER LAG rather than measured once and reused. |
|
|
//| Finite-sample MI bias depends on the realised class counts and the |
|
|
//| bin occupancy, and both move with the lag because different rows |
|
|
//| survive the validity checks - so a single shared floor would be |
|
|
//| the right number for lag 0 and the wrong one everywhere else. |
|
|
//| Cost is the reason it is a REDUCED draw count: a full |
|
|
//| MI_NOISE_PERMUTATIONS sweep at every lag is 200 x historyBars |
|
|
//| scorings. The gate below is deliberately crude for the same |
|
|
//| reason - this profile decides a LOOKBACK, not a trade. |
|
|
//+------------------------------------------------------------------+
|
|
//+------------------------------------------------------------------+
|
|
//| IS "OPTIMAL SL/TP" LEARNABLE? Scores the same features against |
|
|
//| four excursion targets instead of the barrier class. |
|
|
//| |
|
|
//| The question this exists to settle: predicting an optimal stop and |
|
|
//| target decomposes into HOW FAR price travels and WHICH WAY it goes |
|
|
//| first, and those two behave nothing alike. Excursion SIZE is a |
|
|
//| volatility question, and volatility clustering is one of the most |
|
|
//| robust regularities in markets - RANGE is included precisely as a |
|
|
//| positive control that SHOULD clear, and a run where it does not is |
|
|
//| evidence the measurement is broken rather than that the market is |
|
|
//| unpredictable. ASYMMETRY is direction wearing different clothes, |
|
|
//| and it is the only one of the four that can produce expectancy. |
|
|
//| |
|
|
//| So the informative outcome is the CONTRAST, not any single number: |
|
|
//| RANGE/UP/DOWN clearing while ASYM sits at the floor says size is |
|
|
//| predictable and order is not - i.e. the payoff of a predicted |
|
|
//| SL/TP is position sizing and drawdown control, not edge. That is |
|
|
//| worth having under prop-firm limits, and it is not a signal. |
|
|
//| Exit management on RANDOM entries already moved the payoff ratio |
|
|
//| 0.92 -> 5.72 with expectancy FLAT, so this would agree with a test |
|
|
//| that has already been run a different way. |
|
|
//| |
|
|
//| Why this is not answered by the existing verdicts: every MI figure |
|
|
//| this project has produced scored the TRIPLE-BARRIER label, which |
|
|
//| is one specific question ("does the target come before the stop at |
|
|
//| this fixed geometry"). A noise-floor result there says nothing |
|
|
//| about whether excursion MAGNITUDE is learnable - different target, |
|
|
//| different answer, and worth measuring before rebuilding a head. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ReportExcursionInformation(void)
|
|
{
|
|
int targets[] = { MI_TARGET_EXC_RANGE, MI_TARGET_EXC_UP, MI_TARGET_EXC_DOWN, MI_TARGET_EXC_ASYM,
|
|
MI_TARGET_EXC_ASYM_NORM };
|
|
string names[] = { "RANGE up+dn (volatility control)", "UP (MFE)", "DOWN (MAE)",
|
|
"ASYMMETRY up-dn (RAW - confounded by volatility, read the NORM line instead)",
|
|
"ASYMMETRY NORMALISED (up-dn)/(up+dn) (THE ONE THAT MATTERS)" };
|
|
bool asymCleared = false, sizeCleared = false, rawAsymCleared = false;
|
|
for(int k = 0; k < ArraySize(targets); k++)
|
|
{
|
|
//--- Five targets, each with its own full permutation null. Leaving early costs only the targets
|
|
//--- not yet reached; the ones already printed stand, and m_dirEvidence can only have been set by
|
|
//--- a null that ran to completion (see the draws=0 abandon below).
|
|
if(ShutdownRequested())
|
|
return;
|
|
double cols[];
|
|
int labels[];
|
|
int n = BuildMiSample(cols, labels, 0, 0, targets[k]);
|
|
if(n < MI_MIN_SAMPLES)
|
|
{
|
|
Print(ID + ": MI excursion - " + names[k] + ": not enough usable bars to score");
|
|
continue;
|
|
}
|
|
double observed = ScoreMiSample(cols, labels, n, false);
|
|
if(observed < 0.0)
|
|
continue;
|
|
double floorSum = 0.0;
|
|
int draws = 0, atLeast = 0;
|
|
for(int s = 0; s < MI_NOISE_PERMUTATIONS; s++)
|
|
{
|
|
//--- Abandon, never truncate: this null is what sets m_dirEvidence via the NORMALISED asymmetry
|
|
//--- line, and a short null biases p downward - i.e. toward licensing a direction target on a
|
|
//--- test that never finished. draws=0 makes the target below skip cleanly.
|
|
if(ShutdownRequested())
|
|
{
|
|
draws = 0;
|
|
break;
|
|
}
|
|
double sc = ScoreMiSample(cols, labels, n, true);
|
|
if(sc < 0.0)
|
|
continue;
|
|
floorSum += sc;
|
|
if(sc >= observed)
|
|
atLeast++;
|
|
draws++;
|
|
}
|
|
if(draws <= 0)
|
|
continue;
|
|
double floorMean = floorSum / draws;
|
|
double p = (double)(1 + atLeast) / (draws + 1);
|
|
bool clears = (p <= MI_LAG_ALPHA);
|
|
//--- H(Y) is ln(3) by construction (equal-frequency bins), so excess-as-a-share-of-entropy is
|
|
//--- comparable across all four targets and against the barrier label's own figure.
|
|
Print(ID + StringFormat(": MI excursion - %s: %.5f nats/feature vs a block-permuted null of %.5f, "
|
|
"p=%.4f over %d draws%s | %.2f%% of the target's %.3f nats (%d samples)",
|
|
names[k], observed, floorMean, p, draws, (clears ? " <-- CLEARS" : ""),
|
|
100.0 * (observed - floorMean) / MathLog(3.0), MathLog(3.0), n));
|
|
if(targets[k] == MI_TARGET_EXC_ASYM_NORM)
|
|
asymCleared = clears; // the ONLY one a directional claim may rest on
|
|
else
|
|
if(targets[k] == MI_TARGET_EXC_ASYM)
|
|
rawAsymCleared = clears;
|
|
else
|
|
if(clears)
|
|
sizeCleared = true;
|
|
}
|
|
//--- The verdict is the CONTRAST. Spelled out rather than left to be read off five numbers, because
|
|
//--- the wrong reading of "UP clears" is "we can predict profitable trades", and that is precisely
|
|
//--- the inference this report exists to prevent.
|
|
//---
|
|
//--- ORDER MATTERS, and the first version had it wrong: the generic size-not-direction branch was
|
|
//--- tested first, and it is true whenever size clears - i.e. always - so the CONFOUND branch was
|
|
//--- unreachable. Measured 2026-08-07 across three symbols: raw asymmetry cleared on all three while
|
|
//--- normalised collapsed on all three, and the one message that explains why never printed.
|
|
//--- THE SCREEN, not just the commentary. Normalised asymmetry is the only measure in this suite that
|
|
//--- is directional AND free of the volatility confound, so it is the one that can license a direction
|
|
//--- target. Recorded rather than merely printed - see m_dirEvidence.
|
|
if(asymCleared)
|
|
{
|
|
m_dirEvidence = true;
|
|
m_dirEvidenceWhy = "normalised excursion asymmetry cleared its block-permuted null";
|
|
}
|
|
if(asymCleared)
|
|
Print(ID + ": MI excursion VERDICT - NORMALISED ASYMMETRY CLEARS. Scale-free directional "
|
|
"information survives dividing the volatility out, which no barrier-label test has ever "
|
|
"found and which the raw asymmetry could not have established on its own. Before acting: "
|
|
"replicate on instruments NOT used to find it, and check the effect is not concentrated in "
|
|
"one volatility regime. If it holds, this is the first real signal here.");
|
|
else
|
|
if(rawAsymCleared)
|
|
Print(ID + ": MI excursion VERDICT - raw asymmetry cleared but the NORMALISED one did not. That "
|
|
"is the signature of the VOLATILITY CONFOUND, not of direction: up-dn scales with sigma, "
|
|
"so a predictable sigma pushes the value into both outer bins and scores while carrying no "
|
|
"directional content at all - and it does so on every instrument, so replication does not "
|
|
"argue against it. Read the raw line as a restatement of RANGE. Excursion SIZE is "
|
|
"predictable and worth using for position sizing and drawdown control; DIRECTION is not, "
|
|
"so no SL/TP head can create expectancy. Agrees with the random-entry exit test (payoff "
|
|
"ratio 0.92->5.72, expectancy flat).");
|
|
else
|
|
if(sizeCleared)
|
|
Print(ID + ": MI excursion VERDICT - excursion SIZE is predictable, DIRECTION is not. A model "
|
|
"trained to output SL/TP will therefore learn volatility, which is real and useful for "
|
|
"position sizing and drawdown control, but it CANNOT create expectancy: knowing the "
|
|
"next leg spans 3 ATR is worth nothing without knowing which side it spans first. "
|
|
"Agrees with the random-entry exit test (payoff ratio 0.92->5.72, expectancy flat). "
|
|
"Build the head for risk control and stop looking for edge in the exit.");
|
|
else
|
|
Print(ID + ": MI excursion VERDICT - NOTHING clears, INCLUDING the range control. Volatility "
|
|
"clustering is about the most robust regularity in markets, so a range target at the "
|
|
"noise floor points at the measurement, not the market - check the excursion cache "
|
|
"filled and that the sample is not dominated by one volatility regime.");
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
int CExpertSignalAIBase::ReportFeatureLagProfile(void)
|
|
{
|
|
int maxLag = (int)MathMin(MathMax(m_historyBars, 0), MI_LAG_MAX_PROFILE - 1);
|
|
if(maxLag <= 0)
|
|
return 0;
|
|
//--- Per-lag draws retained for the SAME reason the geometry scan retains its own: this report reads a
|
|
//--- profile of ~20 lags, so "does lag k clear ITS OWN null" is the wrong question at every k. See the
|
|
//--- family-wise block below.
|
|
double lagDraws[MI_LAG_MAX_PROFILE][MI_LAG_PERMUTATIONS];
|
|
double lagExcess[MI_LAG_MAX_PROFILE];
|
|
int lagCount[MI_LAG_MAX_PROFILE];
|
|
bool lagValid[MI_LAG_MAX_PROFILE];
|
|
double atZero = 0.0;
|
|
for(int k = 0; k <= maxLag; k++)
|
|
{
|
|
//--- ~21 lags x MI_LAG_PERMUTATIONS full scorings, and the family-wise block below cannot run on a
|
|
//--- partial profile (its bar is the null of the MAXIMUM over lags - drop lags and the maximum is
|
|
//--- taken over a different family). So a stop abandons the whole report and returns "no depth",
|
|
//--- which is what the caller already does with an unmeasurable profile.
|
|
lagValid[k] = false;
|
|
lagExcess[k] = 0.0;
|
|
lagCount[k] = 0;
|
|
if(ShutdownRequested())
|
|
return 0;
|
|
double cols[];
|
|
int labels[];
|
|
int n = BuildMiSample(cols, labels, 0, k);
|
|
if(n < MI_MIN_SAMPLES)
|
|
continue;
|
|
double observed = ScoreMiSample(cols, labels, n, false);
|
|
if(observed < 0.0)
|
|
continue;
|
|
//--- ScoreMiSample shuffles IN PLACE, so the observed statistic must be taken first (above) and the
|
|
//--- draws then reuse the same extracted sample - which is what makes this affordable at all.
|
|
double floorSum = 0.0;
|
|
int draws = 0;
|
|
for(int s = 0; s < MI_LAG_PERMUTATIONS; s++)
|
|
{
|
|
if(ShutdownRequested())
|
|
return 0;
|
|
double sc = ScoreMiSample(cols, labels, n, true);
|
|
if(sc < 0.0)
|
|
continue;
|
|
floorSum += sc;
|
|
lagDraws[k][draws] = sc;
|
|
draws++;
|
|
}
|
|
if(draws < 2)
|
|
continue;
|
|
lagExcess[k] = observed - (floorSum / draws);
|
|
lagCount[k] = draws;
|
|
lagValid[k] = true;
|
|
if(k == 0)
|
|
atZero = lagExcess[k];
|
|
}
|
|
//--- FAMILY-WISE CORRECTION ACROSS LAGS. The first version of this report tested each lag against its
|
|
//--- own null at alpha=0.05 across ~21 lags, which is one expected false positive per run before any
|
|
//--- signal exists - and correlated features make them arrive in CLUSTERS that read like a hump. It
|
|
//--- did exactly that on SP500 H1: 2026-08-06 13:55 starred nothing, 16:22 starred k6/k10/k12/k16 and
|
|
//--- concluded "information survives to lag 16" - same instrument, same 31 features, same 2009 samples,
|
|
//--- while the headline MI moved the other way (p 0.4478 -> 0.8756, observed BELOW its null mean).
|
|
//--- Non-replication on identical data is the signature of an uncorrected multiple comparison.
|
|
//---
|
|
//--- So the bar is the null OF THE MAXIMUM over lags, exactly as the barrier-geometry winner test does
|
|
//--- over candidates: one draw from every lag, keep the largest, repeat. A lag clears only by beating
|
|
//--- that. Draws are centred leave-one-out so each is centred by a mean excluding itself, matching how
|
|
//--- the observed excess is centred. Independence across lags overstates the spread of the maximum
|
|
//--- (neighbouring lags share nearly all their feature window), so this errs toward rejecting.
|
|
int fwDraws = MI_LAG_PERMUTATIONS;
|
|
int validLags = 0;
|
|
for(int k = 0; k <= maxLag; k++)
|
|
if(lagValid[k])
|
|
{
|
|
fwDraws = (int)MathMin(fwDraws, lagCount[k]);
|
|
validLags++;
|
|
}
|
|
double fwMax[MI_LAG_PERMUTATIONS];
|
|
if(validLags <= 0)
|
|
fwDraws = 0;
|
|
for(int s = 0; s < fwDraws; s++)
|
|
{
|
|
double worst = -DBL_MAX;
|
|
for(int k = 0; k <= maxLag; k++)
|
|
{
|
|
if(!lagValid[k])
|
|
continue;
|
|
double sum = 0.0;
|
|
for(int q = 0; q < lagCount[k]; q++)
|
|
sum += lagDraws[k][q];
|
|
double loo = (sum - lagDraws[k][s]) / (lagCount[k] - 1);
|
|
double e = lagDraws[k][s] - loo;
|
|
if(e > worst)
|
|
worst = e;
|
|
}
|
|
fwMax[s] = worst;
|
|
}
|
|
string profile = "";
|
|
int deepest = 0;
|
|
for(int k = 0; k <= maxLag; k++)
|
|
{
|
|
if(!lagValid[k])
|
|
{
|
|
profile += StringFormat(" k%d=n/a", k);
|
|
continue;
|
|
}
|
|
int atLeast = 0;
|
|
for(int s = 0; s < fwDraws; s++)
|
|
if(fwMax[s] >= lagExcess[k])
|
|
atLeast++;
|
|
double pFw = (fwDraws > 0) ? (double)(1 + atLeast) / (fwDraws + 1) : 1.0;
|
|
bool clears = (fwDraws > 0 && pFw <= MI_LAG_ALPHA);
|
|
if(clears)
|
|
deepest = k;
|
|
profile += StringFormat(" k%d=%+.5f%s", k, lagExcess[k], (clears ? "*" : ""));
|
|
}
|
|
Print(ID + StringFormat(": MI feature-lag profile (features from bar i+k, LABEL PINNED to the entry "
|
|
"bar i, so every k is causal; value is excess over that lag's own "
|
|
"block-permutation null; '*' = p<=%.2f against the null of the MAXIMUM over "
|
|
"%d lags, not against the lag's own null - %d lags tested one at a time would "
|
|
"star one per run on noise alone) -%s",
|
|
MI_LAG_ALPHA, validLags, validLags, profile));
|
|
if(deepest <= 0)
|
|
Print(ID + StringFormat(": MI feature-lag profile - NOTHING clears the family-wise null at ANY lag "
|
|
"out to %d bars (entry bar itself %+.5f). The %d-bar lookback is not costing "
|
|
"us information; there is none to lose. This is the blind spot the earlier "
|
|
"reports had: they scored the entry bar alone, so they could not have "
|
|
"distinguished 'no signal anywhere' from 'signal only in the older bars'.",
|
|
maxLag, atZero, maxLag));
|
|
else
|
|
Print(ID + StringFormat(": MI feature-lag profile - information survives to lag %d of %d, clearing "
|
|
"the null of the maximum over %d lags. A lookback shorter than %d would "
|
|
"discard measurable information; a longer one adds input width for none. "
|
|
"BEFORE ACTING ON THIS: re-run it. An uncorrected version of this report "
|
|
"gave opposite answers on two runs over identical data, so one run is not "
|
|
"a result - the shape has to reappear, and ideally on a second instrument.",
|
|
deepest, maxLag, validLags, deepest + 1));
|
|
return deepest;
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ReportBarrierGeometryScan(void)
|
|
{
|
|
//--- SL x1 is deliberately absent: MIN_SL_ATR_MULTIPLIER floors it anyway, and it was rejected on this
|
|
//--- instrument as too tight to survive normal noise. TP grid is exactly the TAKE_PROFIT_MODE enum.
|
|
double slGrid[] = { 2.0, 3.0 };
|
|
int tpGrid[] = { 2, 3, 4, 6, 8, 10 };
|
|
int savedHorizon = m_barrierHorizonBars;
|
|
int barsNow = m_labelCacheBars;
|
|
uint t0 = GetTickCount();
|
|
string rows = "";
|
|
double bestExcess = -1.0;
|
|
string bestName = "";
|
|
int bestSl = 0, bestTp = 0;
|
|
//--- The winner's own horizon, captured while it is in scope. Needed by the DETECTABILITY guard on the
|
|
//--- adoption below: a wider pairing takes longer to resolve, and how long it takes is what decides
|
|
//--- how many INDEPENDENT observations the OOS window can ever yield.
|
|
int bestHorizon = 0;
|
|
//--- Per-candidate null draws, retained so the winner can be tested against the null of the MAXIMUM
|
|
//--- rather than against its own. Only ELIGIBLE candidates are enrolled: the family the maximum was
|
|
//--- actually taken over is the family the gate must correct for, and a disqualified pairing can never
|
|
//--- be the winner however it scores.
|
|
double drawMat[MI_GEOMETRY_MAX_CANDIDATES][MI_GEOMETRY_PERMUTATIONS];
|
|
int drawCount[MI_GEOMETRY_MAX_CANDIDATES];
|
|
int candidates = 0;
|
|
double cfgSl = 0.0, cfgTp = 0.0;
|
|
BarrierMultiples(cfgSl, cfgTp);
|
|
double cfgExcess = -1.0;
|
|
//--- CANDIDATE LIST, built up front so the CONFIGURED pair is scored alongside the grid instead of
|
|
//--- being looked up in it. It was looked up in it until 2026-08-17, and the lookup could not
|
|
//--- succeed: DeriveBarrierGeometry produces CONTINUOUS multiples (3.07 / 1.70) that will never equal
|
|
//--- an integer grid point, so `cfgExcess` stayed at its -1.0 sentinel and the report printed
|
|
//--- "configured 3:2 scores -1.00000" - which reads as a catastrophic score and is actually "never
|
|
//--- evaluated". The one mechanism built to price the shipped geometry could not see it.
|
|
//--- Worse, the skip below excluded target<stop entirely on the grounds that it "inverts the trade's
|
|
//--- whole premise" - while the derivation was shipping exactly that. The grid keeps the skip (an
|
|
//--- inverted ENUM pairing is still not something to crown), but the incumbent is always scored.
|
|
double candSl[], candTp[];
|
|
bool candIsCfg[];
|
|
int candN = 0;
|
|
ArrayResize(candSl, ArraySize(slGrid) * ArraySize(tpGrid) + 1);
|
|
ArrayResize(candTp, ArraySize(slGrid) * ArraySize(tpGrid) + 1);
|
|
ArrayResize(candIsCfg, ArraySize(slGrid) * ArraySize(tpGrid) + 1);
|
|
for(int a = 0; a < ArraySize(slGrid); a++)
|
|
for(int b = 0; b < ArraySize(tpGrid); b++)
|
|
{
|
|
if((double)tpGrid[b] < slGrid[a])
|
|
continue;
|
|
candSl[candN] = slGrid[a];
|
|
candTp[candN] = (double)tpGrid[b];
|
|
candIsCfg[candN] = false;
|
|
candN++;
|
|
}
|
|
if(cfgSl > 0.0 && cfgTp > 0.0)
|
|
{
|
|
candSl[candN] = cfgSl;
|
|
candTp[candN] = cfgTp;
|
|
candIsCfg[candN] = true;
|
|
candN++;
|
|
}
|
|
m_barrierScanLiveLabels = true;
|
|
{
|
|
for(int c = 0; c < candN; c++)
|
|
{
|
|
//--- THE HEAVIEST SCAN IN THE EA: every pairing relabels the whole sampled history and then draws
|
|
//--- MI_GEOMETRY_PERMUTATIONS nulls off it. Break rather than return, so the four scan fields are
|
|
//--- always put back by the single restore below the loop - m_barrierScanLiveLabels left true
|
|
//--- makes ComputeLabelForBar read the LAST CANDIDATE's multiples as if they were the configured
|
|
//--- geometry, which would silently relabel on a pairing nothing chose.
|
|
if(ShutdownRequested())
|
|
break;
|
|
m_barrierScanSlMult = candSl[c];
|
|
m_barrierScanTpMult = candTp[c];
|
|
m_barrierHorizonBars = ComputeBarrierHorizonBars(barsNow);
|
|
bool clamped = m_barrierHorizonClamped;
|
|
m_barrierScanTimeouts = 0;
|
|
double gc[];
|
|
int gl[];
|
|
int gn = BuildMiSample(gc, gl);
|
|
if(gn < MI_MIN_SAMPLES)
|
|
continue;
|
|
double obs = ScoreMiSample(gc, gl, gn, false);
|
|
//--- Class shares of THIS geometry's label, so a geometry that scores well by having almost
|
|
//--- nothing left to predict is visible as such instead of winning quietly.
|
|
int cB = 0, cS = 0;
|
|
for(int q = 0; q < gn; q++)
|
|
{
|
|
if(gl[q] == 0)
|
|
cB++;
|
|
else
|
|
if(gl[q] == 1)
|
|
cS++;
|
|
}
|
|
double dirShare = 100.0 * (cB + cS) / gn;
|
|
double timeoutShare = 100.0 * m_barrierScanTimeouts / gn;
|
|
//--- The MIN REWARD:RISK test that used to gate enrolment here is GONE (2026-08-09) along with
|
|
//--- Min_Risk_Reward_Ratio itself. Its purpose was to stop the scan crowning a geometry the live
|
|
//--- rejection filter would then throw every setup away at - but with no rejection filter there
|
|
//--- is nothing to collide with, and excluding low-ratio pairings was excluding them on a rule
|
|
//--- rather than on the measurement. CLAMPING remains disqualifying, and for an unrelated reason
|
|
//--- that still holds: a clamped label describes a trade truncated by the horizon rather than
|
|
//--- resolved at SL/TP, so it is not the target the EA would actually hold to.
|
|
bool eligible = !clamped;
|
|
//--- These draws now serve two purposes. Per candidate they still centre the printed score. Across
|
|
//--- candidates they form the null of the maximum, which is the only thing that can say whether the
|
|
//--- WINNER is real - so they are retained rather than reduced to a mean and discarded.
|
|
double nullSum = 0.0;
|
|
int nd = 0;
|
|
for(int s = 0; s < MI_GEOMETRY_PERMUTATIONS; s++)
|
|
{
|
|
//--- Break only: the outer loop's check runs next and does the restore-and-return in one
|
|
//--- place. This candidate's partial draws are discarded with everything else, so a short
|
|
//--- null can never reach the family-wise gate below.
|
|
if(ShutdownRequested())
|
|
break;
|
|
double sc = ScoreMiSample(gc, gl, gn, true);
|
|
if(sc < 0.0)
|
|
continue;
|
|
nullSum += sc;
|
|
if(eligible && candidates < MI_GEOMETRY_MAX_CANDIDATES)
|
|
drawMat[candidates][nd] = sc;
|
|
nd++;
|
|
}
|
|
if(eligible && candidates < MI_GEOMETRY_MAX_CANDIDATES)
|
|
{
|
|
drawCount[candidates] = nd;
|
|
candidates++;
|
|
}
|
|
double nullMean = (nd > 0) ? nullSum / nd : -1.0;
|
|
double excess = (nullMean >= 0.0) ? (obs - nullMean) : 0.0;
|
|
//--- Base rate m/(m+k) IS the break-even win rate at this geometry - print it so the ranking is
|
|
//--- read next to the bar the model would have to clear, not in isolation.
|
|
double breakeven = 100.0 * candSl[c] / (candSl[c] + candTp[c]);
|
|
//--- Ranking is now purely the measurement: every unclamped pairing competes, whatever its
|
|
//--- reward:risk. Note the break-even printed alongside is what makes a low-ratio winner
|
|
//--- readable rather than alarming - 1:1 needs 50% precision where 1:3 needs 25%, and the
|
|
//--- scan's value column is already excess information over that geometry's OWN null, so the
|
|
//--- comparison across pairings is like-for-like.
|
|
string name = candIsCfg[c] ? StringFormat("CFG %.2f:%.2f", candSl[c], candTp[c])
|
|
: StringFormat("%.0f:%.0f", candSl[c], candTp[c]);
|
|
rows += StringFormat("%s%s(h%d%s,be%.0f%%,dir%.0f%%,to%.0f%%)=%+.5f", (rows == "" ? "" : " "),
|
|
name, m_barrierHorizonBars, (clamped ? "!" : ""), breakeven,
|
|
dirShare, timeoutShare, excess);
|
|
//--- Only unclamped, tradeable geometries are eligible to WIN - see the header. The rest are
|
|
//--- still printed, so a disqualification is visible rather than a silent omission.
|
|
//--- The CONFIGURED pair is scored for comparison but never crowned: it is already in force, and
|
|
//--- its continuous multiples are not an enum pairing this scan could adopt anyway.
|
|
if(eligible && !candIsCfg[c] && excess > bestExcess)
|
|
{
|
|
bestExcess = excess;
|
|
bestName = name;
|
|
//--- The grid values ARE the enum values (SL_ATR_x2 == 2, TP_ATR_x8 == 8), so the winning
|
|
//--- pairing can be adopted directly with no lookup table to drift out of step.
|
|
bestSl = (int)candSl[c];
|
|
bestTp = (int)candTp[c];
|
|
bestHorizon = m_barrierHorizonBars;
|
|
}
|
|
if(candIsCfg[c])
|
|
cfgExcess = excess;
|
|
}
|
|
}
|
|
m_barrierScanLiveLabels = false;
|
|
m_barrierScanSlMult = 0.0;
|
|
m_barrierScanTpMult = 0.0;
|
|
m_barrierHorizonBars = savedHorizon;
|
|
//--- THE ONE PLACE an abandoned scan leaves from, after the restore above and before anything reads what
|
|
//--- it collected. A stop can truncate the last candidate's null, and fwDraws below is the MINIMUM draw
|
|
//--- count across candidates - so one short candidate shrinks the null of the maximum for all of them and
|
|
//--- would let a pairing be ADOPTED on a null that never finished. Nothing here is worth that, and
|
|
//--- nothing here is worth the teardown budget either.
|
|
if(ShutdownRequested())
|
|
{
|
|
PrintFormat("%s: barrier-geometry scan ABANDONED - stop requested. Scan state restored; the geometry"
|
|
" in force is unchanged.", ID);
|
|
return;
|
|
}
|
|
Print(ID + StringFormat(": barrier-geometry scan (SL:TP; h=horizon, '!'=CLAMPED and disqualified - a clamped "
|
|
"label truncates a trade the EA would hold to SL/TP; be=break-even win rate, "
|
|
"dir=%%bars with a tradeable direction, to=%%timed out; value is entry-time "
|
|
"information in nats above that geometry's own null; 'CFG' is the geometry "
|
|
"actually in force, scored as a peer but never crowned) - %s | configured "
|
|
"%.2f:%.2f scores %+.5f, best eligible GRID pairing is %s at %+.5f (%.1fs)",
|
|
rows, cfgSl, cfgTp, cfgExcess, (bestName == "" ? "none" : bestName), bestExcess,
|
|
(GetTickCount() - t0) / 1000.0));
|
|
//--- FAMILY-WISE GATE. bestExcess is a MAXIMUM over the eligible candidates, and the maximum of several
|
|
//--- draws from a null sits well above any single draw from it - so testing the winner against its own
|
|
//--- null asks the wrong question and will crown a winner on pure noise almost every time. What follows
|
|
//--- rebuilds the null OF THE MAXIMUM: take one permutation draw from every candidate, keep the largest,
|
|
//--- repeat. bestExcess then has to beat that distribution, not a single-candidate one.
|
|
//---
|
|
//--- The draws are centred LEAVE-ONE-OUT so the comparison is like for like: the observed score is
|
|
//--- centred by draws that do not contain it, so each draw must be too. Centring a draw by a mean that
|
|
//--- includes it shrinks it toward zero, which would deflate the null and let the winner through.
|
|
//---
|
|
//--- Draws are independent across candidates here while the real ones are correlated (the candidates
|
|
//--- share features and heavily overlapping label windows). Independence makes the maximum MORE spread
|
|
//--- out than the truth, so the gate errs toward rejecting - the safe direction when passing costs a
|
|
//--- full relabel and retrain of every topology.
|
|
double pFamily = 1.0;
|
|
int fwDraws = 0;
|
|
double nullMaxSum = 0.0;
|
|
int nullMaxCount = 0;
|
|
if(candidates > 0)
|
|
{
|
|
fwDraws = MI_GEOMETRY_PERMUTATIONS;
|
|
for(int c = 0; c < candidates; c++)
|
|
fwDraws = (int)MathMin(fwDraws, drawCount[c]);
|
|
int atLeast = 0;
|
|
for(int s = 0; s < fwDraws; s++)
|
|
{
|
|
double worst = -DBL_MAX;
|
|
for(int c = 0; c < candidates; c++)
|
|
{
|
|
if(drawCount[c] < 2)
|
|
continue;
|
|
double sum = 0.0;
|
|
for(int q = 0; q < drawCount[c]; q++)
|
|
sum += drawMat[c][q];
|
|
double loo = (sum - drawMat[c][s]) / (drawCount[c] - 1);
|
|
double e = drawMat[c][s] - loo;
|
|
if(e > worst)
|
|
worst = e;
|
|
}
|
|
if(worst > -DBL_MAX && worst >= bestExcess)
|
|
atLeast++;
|
|
//--- WINNER'S-CURSE PENALTY, accumulated from the same draws the p-value uses. `worst` is the
|
|
//--- MAXIMUM excess over all candidates in a pure-noise draw, so its mean across draws is
|
|
//--- exactly what a best-of-K selection is expected to report when there is nothing there.
|
|
//--- Measured rather than assumed: this is the empirical form of the sqrt(2 ln K) x SE penalty
|
|
//--- the SQX EdgeFinder plugin applies to every maximum it reports (Stats.java:79-88), and it
|
|
//--- needs no normality assumption because the permutation draws ARE the null distribution.
|
|
if(worst > -DBL_MAX)
|
|
{
|
|
nullMaxSum += worst;
|
|
nullMaxCount++;
|
|
}
|
|
}
|
|
pFamily = (fwDraws > 0) ? (double)(1 + atLeast) / (fwDraws + 1) : 1.0;
|
|
}
|
|
//--- The winner's excess, SHRUNK toward zero by that penalty. A best-of-K maximum is biased upward by
|
|
//--- construction - it is the largest of K noisy draws - so the raw number overstates what adopting it
|
|
//--- will actually deliver, and it is the number the adoption message quotes and compares against the
|
|
//--- incumbent. Clearing the family-wise gate says the ranking is real; it does NOT say the effect is
|
|
//--- as large as the top row reads. James-Stein form: effect x max(0, 1 - penalty^2/effect^2), which
|
|
//--- leaves a large effect nearly untouched and collapses a marginal one to ~0.
|
|
double nullMaxMean = (nullMaxCount > 0) ? nullMaxSum / nullMaxCount : 0.0;
|
|
double shrunkExcess = bestExcess;
|
|
if(bestExcess > 0.0 && nullMaxMean > 0.0)
|
|
shrunkExcess = bestExcess * MathMax(0.0, 1.0 - (nullMaxMean * nullMaxMean) / (bestExcess * bestExcess));
|
|
bool winnerReal = (bestName != "" && bestExcess > 0.0 && fwDraws > 0 && pFamily <= MI_GEOMETRY_ALPHA);
|
|
Print(ID + StringFormat(": barrier-geometry winner test - %s at %+.5f is the best of %d ELIGIBLE "
|
|
"candidates, so it is tested against the null of the maximum over %d, not its "
|
|
"own: p=%.4f over %d draws (need <=%.2f). SHRUNK effect %+.5f (a best-of-K maximum is "
|
|
"biased upward by construction; the winner's-curse penalty here is the MEAN "
|
|
"noise-draw maximum %+.5f, measured on these same draws - plan on the shrunk "
|
|
"number, not the raw one). %s", (bestName == "" ? "none" : bestName),
|
|
bestExcess, candidates, candidates, pFamily, fwDraws, MI_GEOMETRY_ALPHA, shrunkExcess, nullMaxMean,
|
|
(winnerReal ? "CLEARS - the ranking is not selection noise."
|
|
: "DOES NOT CLEAR - a max this large happens routinely when every candidate is "
|
|
"pure noise, so the ranking carries no information and the top row is not a "
|
|
"finding. Change nothing.")));
|
|
//--- ADOPT, don't advise. SL_Mode/TP_Mode stopped being inputs on 2026-08-07, so this scan is now the
|
|
//--- thing that chooses the barrier - which is exactly why the family-wise gate above had to exist
|
|
//--- first. Three conditions, all necessary:
|
|
//--- winnerReal - it beat the null of the MAXIMUM, not merely the incumbent and not merely zero.
|
|
//--- m_eraCount==0 - relabelling a partly-trained net would move the target out from under weights
|
|
//--- already fitted to the old one. Same gate the indicator tuner uses.
|
|
//--- != current - nothing to do when the measurement agrees with the default.
|
|
//--- A model that already exists never reaches here with anything to change: its geometry is pinned in
|
|
//--- the .cfg and adopted at load, so the pairing a run trains on is the pairing it keeps.
|
|
//--- DETECTABILITY GUARD ON THE ADOPTION, and it exists because making this scan authoritative
|
|
//--- (2026-08-17) put two objectives in charge of one decision without reconciling them. The scan
|
|
//--- maximises entry-time INFORMATION in nats. The deploy gate needs enough INDEPENDENT observations
|
|
//--- to certify an edge. Those are not the same quantity and they pull opposite ways: a wider pairing
|
|
//--- carries more information per call AND takes longer to resolve, and overlapping labels are worth
|
|
//--- ~1/L each, so tripling the horizon divides the independent sample by ~3 and multiplies the SE the
|
|
//--- gate must beat by ~sqrt(3). USDJPY's winner today is 2:8 at h192 against an incumbent 1.61:3.21
|
|
//--- at h64 - exactly that trade, made silently.
|
|
//---
|
|
//--- Conservative by construction: the horizon is an UPPER bound on the mean label lifespan, so
|
|
//--- oosBars/horizon is a LOWER bound on the independent observations available. If even that bound
|
|
//--- cannot support a generous edge, no model trained on this pairing could ever be certified on this
|
|
//--- window - which makes adopting it a guaranteed dead end rather than a promising direction.
|
|
bool winnerDetectable = true;
|
|
double needIndep = 0.0, haveIndep = 0.0;
|
|
if(bestSl > 0 && bestTp > 0 && bestHorizon > 0)
|
|
{
|
|
double p = (double)bestSl / ((double)bestSl + (double)bestTp); // break-even at this pairing
|
|
double d = ADOPT_MIN_DETECTABLE_EDGE; // the edge we would deploy on
|
|
needIndep = (EDGE_MIN_SIGMAS * EDGE_MIN_SIGMAS) * p * (1.0 - p) / (d * d);
|
|
haveIndep = (double)barsNow * (MathMax(0, MathMin(100, m_oosSplitPct)) / 100.0) / (double)bestHorizon;
|
|
winnerDetectable = (haveIndep >= needIndep);
|
|
}
|
|
if(winnerReal && !winnerDetectable && m_eraCount == 0)
|
|
Print(ID + StringFormat(": barrier-geometry winner %s NOT ADOPTED - it wins on information (%+.5f"
|
|
" nats) and loses on DETECTABILITY. At break-even %.0f%% it needs %.0f"
|
|
" independent calls to certify a %.0fpp edge, and its %d-bar horizon leaves"
|
|
" this OOS window at most %.0f - so no win rate a model could reach would"
|
|
" ever clear the deploy gate on this pairing. Keeping the incumbent. More"
|
|
" information per trade is worth nothing if it buys too few independent"
|
|
" trades to prove: WIDTH is not free, and this is the constraint that"
|
|
" decides, not the nats.",
|
|
bestName, bestExcess,
|
|
100.0 * (double)bestSl / ((double)bestSl + (double)bestTp),
|
|
needIndep, 100.0 * ADOPT_MIN_DETECTABLE_EDGE, bestHorizon, haveIndep));
|
|
if(winnerReal && winnerDetectable && m_eraCount == 0 && bestSl > 0 && bestTp > 0
|
|
&& (bestSl != m_sl_mode || bestTp != m_tp_mode))
|
|
{
|
|
Print(ID + StringFormat(": adopting barrier geometry %s - it carries %+.5f nats of entry-time "
|
|
"information against the configured %.0f:%.0f's %+.5f, and cleared the "
|
|
"family-wise gate. Relabelling and training on it. Chance precision equals "
|
|
"break-even at EVERY geometry, so this does not hand us expectancy; it puts "
|
|
"more of the answer inside the features' reach, which is the one thing no "
|
|
"change of topology can do.", bestName, bestExcess, cfgSl, cfgTp, cfgExcess));
|
|
//--- AND INTO THE PAIR THAT ACTUALLY GOVERNS, which the mode ints have not been since the derived
|
|
//--- geometry landed. BarrierMultiples() ranks DERIVED above the modes, so on any model carrying a
|
|
//--- derived pair - which is every model with a .cfg, including a fresh one whose weights are gone
|
|
//--- but whose sidecar survived - setting the modes alone changed NOTHING. The scan announced
|
|
//--- "adopting 2:8 ... relabelling and training on it" and the very next line labelled at
|
|
//--- 1.61:3.21 (USDJPY, 2026-08-17 14:24:12.844 vs .979). Two measurements choosing the same thing,
|
|
//--- one of them silently inert, and a log line that stated the opposite of what happened.
|
|
//--- ONE AUTHORITY: the derived pair. It is what the labels read, what the deploy gate certifies,
|
|
//--- what g_Derived*AtrMult places on the live order, and what the .cfg pins across restarts. The
|
|
//--- mode ints are legacy (SL_Mode/TP_Mode stopped being inputs on 2026-08-07) and are kept in step
|
|
//--- above only so the fallback path and the fingerprint stay coherent.
|
|
ApplyAdoptedGeometry((double)bestSl, (double)bestTp, bestSl, bestTp);
|
|
//--- PUBLISH IT TO THE CHART. The MI chain that ends in this scan runs once per chart, so the other
|
|
//--- three members never measure this and would otherwise keep labelling on their own derived pair
|
|
//--- while this one relabels - four members, two targets, one averaged vote. See the measurement in
|
|
//--- g_ensembleChartGeomAdopted's comment.
|
|
if(m_ensembleMember)
|
|
{
|
|
g_ensembleChartGeomAdopted = true;
|
|
g_ensembleChartGeomSl = m_derivedSlMult;
|
|
g_ensembleChartGeomTp = m_derivedTpMult;
|
|
g_ensembleChartGeomSlMode = m_sl_mode;
|
|
g_ensembleChartGeomTpMode = m_tp_mode;
|
|
}
|
|
}
|
|
else
|
|
if(winnerReal && m_eraCount > 0 && bestSl > 0 && (bestSl != m_sl_mode || bestTp != m_tp_mode))
|
|
Print(ID + ": barrier-geometry scan prefers " + bestName + ", but this model is already trained "
|
|
"(era " + IntegerToString(m_eraCount) + "). Its geometry is pinned to what it learned; "
|
|
"delete the weights if you want it re-measured.");
|
|
else
|
|
if(bestName == "")
|
|
Print(ID + ": barrier-geometry scan - every geometry with a long enough horizon was "
|
|
"disqualified or scored at zero. Nothing here to switch to; the limit is the feature "
|
|
"set, not the target.");
|
|
}
|
|
//+------------------------------------------------------------------+
|
|
//| Outer loop around Train(). Tuning is now a one-shot filter pass |
|
|
//| that runs BEFORE the first era and costs seconds, so this is a |
|
|
//| straight pass-through to Train() on every later call. |
|
|
//+------------------------------------------------------------------+
|
|
//+------------------------------------------------------------------+
|
|
//| See the declaration. Every side effect an adopted geometry must |
|
|
//| carry, in ONE place, because it has two callers: the member whose |
|
|
//| scan chose the pairing, and the siblings that learn it from |
|
|
//| g_ensembleChartGeomAdopted without ever running a scan. |
|
|
//+------------------------------------------------------------------+
|
|
void CExpertSignalAIBase::ApplyAdoptedGeometry(double sl, double tp, int slMode, int tpMode)
|
|
{
|
|
//--- Legacy mode ints first. SL_Mode/TP_Mode stopped being inputs on 2026-08-07 and no longer decide
|
|
//--- anything, but the fallback path in BarrierMultiples and the config fingerprint still read them, so
|
|
//--- leaving them behind the derived pair is how the two disagree.
|
|
m_sl_mode = slMode;
|
|
m_tp_mode = tpMode;
|
|
//--- Same floor DeriveBarrierGeometry applies, for the same reason: OpenParams widens any stop tighter
|
|
//--- than this, and a live stop wider than the labelled one grades the model on a bet it is not placing.
|
|
if(sl < MIN_SL_ATR_MULTIPLIER)
|
|
sl = MIN_SL_ATR_MULTIPLIER;
|
|
m_derivedSlMult = sl;
|
|
m_derivedTpMult = tp;
|
|
m_geometryDerived = true;
|
|
//--- LATCHED, or the very next label prebuild undoes this. DeriveBarrierGeometry re-measures the scale
|
|
//--- from the excursion quantiles on every era-0 pass and overwrites the pair - and this sets
|
|
//--- m_labelCachePrebuilt = false below, which is exactly what triggers that pass. Without the latch the
|
|
//--- sequence is: adopt 2:8, invalidate the cache, re-derive 1.61:3.21, train on 1.61:3.21, and print
|
|
//--- that it adopted 2:8.
|
|
m_geometryAdopted = true;
|
|
//--- Republished immediately, not at the next era end: between here and there the label cache is rebuilt
|
|
//--- under the adopted pair, and a live order placed in that window would otherwise carry the superseded
|
|
//--- geometry.
|
|
g_DerivedSlAtrMult = m_derivedSlMult;
|
|
g_DerivedTpAtrMult = m_derivedTpMult;
|
|
//--- The sidecar holds the OLD pair; force it to be rewritten so a restart resumes on the adopted one
|
|
//--- rather than silently reverting to what this just replaced.
|
|
m_geometryCfgSaved = false;
|
|
Print(ID + StringFormat(": geometry authority - stop %.2f*ATR / target %.2f*ATR is now the pair the"
|
|
" labels, the deploy gate and the live order ALL read. The SL_Mode/TP_Mode"
|
|
" ints are kept in step but no longer decide anything.",
|
|
m_derivedSlMult, m_derivedTpMult));
|
|
//--- The cache holds labels computed under the OLD barriers, so it has to be discarded rather than
|
|
//--- appended to - Train()'s !m_labelCachePrebuilt gate then rebuilds it under the adopted pair before
|
|
//--- era 0 starts.
|
|
m_labelCachePrebuilt = false;
|
|
ArrayInitialize(m_labelCacheHasValue, false);
|
|
//--- AND UNLATCH THE HORIZON, which is otherwise resolved once per process and held. Adopting a wider
|
|
//--- target without this labels the new geometry against the OLD ceiling - 2:8 wants ~192 bars and would
|
|
//--- silently get 2:6's 128 - which is precisely the truncation that made every model learn "target
|
|
//--- within 128 bars" while the EA holds to SL/TP (fixed 2026-08-01 in 168422f). The truncation lands in
|
|
//--- Neutral, not in the timeout counter that watches for it, so it does not announce itself.
|
|
//--- EnsureBarrierHorizon() re-derives and re-logs on the next call.
|
|
m_barrierHorizonResolved = false;
|
|
}
|
|
void CExpertSignalAIBase::TuneIndicatorsAndTrain(datetime StartTrainBar = 0)
|
|
{
|
|
//--- FIRST STATEMENT IN THE WHOLE TRAINING ENTRY POINT, ahead of every latch below it (m_tuneFilterDone,
|
|
//--- g_ensembleChartTuneDone) so a stop cannot mark a sweep as "already run" without running it. The
|
|
//--- individual scans yield on ShutdownRequested() as well; this simply refuses to start the chain.
|
|
if(ShutdownRequested())
|
|
return;
|
|
//--- Publish the caller's window anchor so StartLabelCachePrebuild() sizes its window with the SAME
|
|
//--- expression Train() uses. This assignment existed before the God-class split and was dropped by
|
|
//--- it, leaving m_tuneStartTrainBar permanently 0 - harmless while nothing read it, and a real defect
|
|
//--- the moment 0c85c54 made the prebuild reset dtStudied from it: Train() then computed
|
|
//--- max(StartTrainBar, windowFloor) while the prebuild computed max(0, windowFloor), so the two
|
|
//--- disagreed about `bars`, EnsureBarCachesCapacity() saw a changed size every era start, and each
|
|
//--- era immediately re-armed a full 38k-bar prebuild instead of training.
|
|
m_tuneStartTrainBar = StartTrainBar;
|
|
bool anyTunable = (m_useADCumulativeDelta || m_useADShorteningOfThrust || m_useADWyckoffEventStream ||
|
|
m_useADWyckoffFailedStructure || m_useADWyckoffSignificantBarInversion ||
|
|
m_useMA || m_useRSI || m_useMACD || m_useIchimoku);
|
|
//--- Tune once per fresh model, before any weight has been trained. Gated on m_labelCachePrebuilt
|
|
//--- because the score needs labels, and on era 0 because re-tuning a partly-trained network would
|
|
//--- change its inputs out from under weights already fitted to the old ones.
|
|
if(m_autoTuneIndicators && anyTunable && !m_tuneFilterDone && m_labelCachePrebuilt && m_eraCount == 0)
|
|
{
|
|
m_tuneFilterDone = true;
|
|
if(m_ensembleMember && g_ensembleChartTuneDone)
|
|
{
|
|
//--- Another member on this chart already ran the identical sweep - apply its outcome instead
|
|
//--- of recomputing it (see g_ensembleChartTuneDone at the top of this file).
|
|
//--- SAY IT ON THE PANEL, not only in the journal. Until now this branch printed and published
|
|
//--- nothing, so an ensemble warm-up showed ONE row reading "scoring indicator settings..." for
|
|
//--- a minute while the other three sat on a stale line - which reads as three members doing
|
|
//--- nothing, rather than three members correctly declining to redo a chart-wide measurement.
|
|
PublishStatus(ID + " : adopting the chart's tuned indicators...");
|
|
//--- THE PARAMETERS are adopted only when a winner was installed...
|
|
if(g_ensembleChartTuneInstalled)
|
|
m_indicatorTuner.Unflatten(g_ensembleChartTuneSettings);
|
|
//--- ...but the HANDLES must be rebuilt EITHER WAY, and that is not a tidiness point - it is the
|
|
//--- cause of the "silent block failure" that cost six sessions. MEASURED 2026-08-17, twice:
|
|
//--- USDJPY 18:10:03 PAI sweeps -> 18:10:29.86/.91/.95 CONV/LSTM/HYB take this branch
|
|
//--- (winner REJECTED, so the old code re-inited NOTHING)
|
|
//--- -> 18:10:30.06/.07/.07 all three: MA=-1(h13), every sweep bar rejected
|
|
//--- XAUUSD 18:10:11 -> 18:10:42.19/.23/.27 -> 18:10:42.33 - identical, same 100 ms gap
|
|
//--- ALL TWELVE dead handles across both charts and all four members reported the SAME NUMBER,
|
|
//--- h13. That is the whole finding: it was never four handles, it was ONE. MT5 refcounts
|
|
//--- indicator requests, so four members asking for the same iMA on the same symbol/period get
|
|
//--- one shared handle - and TuneIndicatorsByFilter creates and drops ~35 of them scoring
|
|
//--- candidates. When it restores the configured settings the runner ends up holding a live
|
|
//--- handle while its siblings still hold the number the terminal has already freed.
|
|
//--- The old guard made re-initialising conditional on the winner being INSTALLED, which is
|
|
//--- exactly backwards: a REJECTED sweep churns just as many handles, and every one of the
|
|
//--- twelve recoveries above followed a rejection. Four handle creations per chart is nothing.
|
|
ReInitADIndicators(m_indicatorsPtr);
|
|
RefreshData();
|
|
Print(ID + ": indicator auto-tune already ran on this chart - same indicators, same features, "
|
|
"same labels, same answer. " +
|
|
(g_ensembleChartTuneInstalled
|
|
? "Adopting the installed winner so every member trains on the same feature vector."
|
|
: "Keeping the configured settings (the sweep's winner was rejected by the selection gate).") +
|
|
" The first member's auto-tune report above is this model's too.");
|
|
}
|
|
else
|
|
{
|
|
//--- Names the SCOPE, because the scope is what the other rows' silence means. This sweep is one
|
|
//--- measurement of one chart's indicators against one chart's labels, so exactly one member runs
|
|
//--- it and the rest adopt the answer - saying "for the whole chart" here is what tells the
|
|
//--- operator that three idle rows are the design and not a stall.
|
|
PublishStatus(ID + (m_ensembleMember
|
|
? " : scoring indicator settings for the whole chart..."
|
|
: " : scoring indicator settings..."));
|
|
//--- Snapshot the configured settings first: "did the sweep install?" is answered by comparing
|
|
//--- against the final settings, since a rejected winner is restored to exactly these values.
|
|
double tuneCfgBefore[];
|
|
m_indicatorTuner.Flatten(tuneCfgBefore);
|
|
TuneIndicatorsByFilter();
|
|
if(m_ensembleMember)
|
|
{
|
|
m_indicatorTuner.Flatten(g_ensembleChartTuneSettings);
|
|
g_ensembleChartTuneInstalled = false;
|
|
for(int tp = 0; tp < ArraySize(tuneCfgBefore); tp++)
|
|
if(g_ensembleChartTuneSettings[tp] != tuneCfgBefore[tp])
|
|
{
|
|
g_ensembleChartTuneInstalled = true;
|
|
break;
|
|
}
|
|
g_ensembleChartTuneDone = true;
|
|
//--- The sweep ends in ReportFeatureLabelInformation(), so the chart-level MI report is
|
|
//--- done too - mark it, or every other member would rerun the ~200-draw nulls the MI
|
|
//--- gate below exists to save.
|
|
if(m_miReportDone)
|
|
g_ensembleChartMiReportDone = true;
|
|
}
|
|
}
|
|
//--- the winning parameters change the input vector, so the network must start from scratch on it
|
|
BuildFreshTopology();
|
|
}
|
|
//--- The DIAGNOSTIC half runs even when the sweep does not: on a resumed model, on one whose tuner is
|
|
//--- switched off, and on one with nothing tunable. It reads the cached features and writes nothing,
|
|
//--- so none of the reasons the sweep is gated apply to it - and tying it to that gate meant the only
|
|
//--- way to see the answer on a running model was to delete the model.
|
|
//---
|
|
//--- THAT INTENT WAS NOT ACHIEVED UNTIL 2026-08-07. Moving the diagnostic out of the tuner's gate
|
|
//--- left it behind m_labelCachePrebuilt, which has exactly the same effect: the eager label pre-scan
|
|
//--- runs only on a FRESH start, because a resumed net labels lazily per bar (see the "skipped
|
|
//--- entirely when a trained net was loaded from disk" note in the prebuild). So on a resumed model
|
|
//--- the flag is false forever and the entire MI block - headline, positive control, alignment scan,
|
|
//--- lag profile, geometry scan, winner test - silently never ran. Measured on SP500 H1 2026-08-07:
|
|
//--- attached at era 271, still nothing by era 314, and every diagnostic captured on 08-05/06 came
|
|
//--- immediately after a weights reset. The only way to see the answer was still to delete the model.
|
|
//---
|
|
//--- So drive the prebuild ourselves when it is the only thing missing. It is safe on a trained net:
|
|
//--- its one fresh-net side effect, pushing the output-layer bias toward the dominant class, is
|
|
//--- already gated on m_eraCount == 0, and the scan itself only fills label caches. Train()'s own
|
|
//--- m_labelPrebuildActive gate advances it to completion, so this costs one short deferral (~1s at
|
|
//--- 38k bars) on the first attach and nothing afterwards.
|
|
//---
|
|
//--- NOT sampled from the lazily-filled cache instead: BuildMiSample skips bars that carry no cached
|
|
//--- label, so on a resumed model it would quietly score whichever subset training happened to have
|
|
//--- visited. That is a biased subsample presented as a measurement - the failure mode this whole
|
|
//--- diagnostic exists to catch.
|
|
else if(!m_miReportDone && !m_labelCachePrebuilt && !m_labelPrebuildActive)
|
|
{
|
|
//--- Announce only on a start that actually took. StartLabelCachePrebuild() returns without arming
|
|
//--- if the buffers/history are not ready yet and is simply retried on the next call, so printing
|
|
//--- unconditionally would repeat the line once per bar event until it succeeds.
|
|
StartLabelCachePrebuild();
|
|
//--- Says WHICH case this is rather than asserting the resumed one. The first version claimed
|
|
//--- "resumed from disk" unconditionally, and then printed it above a "seeding era 0" line on a
|
|
//--- brand-new model - the branch fires whenever the cache is not built, which is equally true
|
|
//--- before a fresh model's first prebuild. A diagnostic that misreports its own trigger is worse
|
|
//--- than one that says nothing, because it gets quoted back as evidence.
|
|
if(m_labelPrebuildActive)
|
|
Print(ID + (m_modelLoadedFromDisk
|
|
? ": MI diagnostics need a complete label cache and this model resumed from disk "
|
|
"(labels are filled lazily, so the cache covers only the bars training has "
|
|
"visited) - running the one-time pre-scan now, then the report. Training resumes "
|
|
"where it left off."
|
|
: ": MI diagnostics need a complete label cache and this model has not built one yet "
|
|
"- running the pre-scan now, then the report."));
|
|
}
|
|
else if(!m_miReportDone && m_labelCachePrebuilt)
|
|
{
|
|
//--- WAIT FOR THE CROSS-ASSET PANEL. It is part of the feature vector but it is built inside
|
|
//--- Train(), so on a fresh run this diagnostic would otherwise describe a NARROWER vector than
|
|
//--- the one training goes on to use. Observed 2026-08-02 on SP500 H1: the MI report, the
|
|
//--- alignment scan and the barrier-geometry scan all ran at 00:41:25, while the panel first
|
|
//--- built successfully at 01:12:47 - so every number they printed, including the geometry scan
|
|
//--- that is supposed to CHOOSE the training target, was measured on a feature set training
|
|
//--- never saw. Train() rebuilds the panel each era, so simply deferring lands the report on an
|
|
//--- era where the vector is complete.
|
|
//--- Never wait forever: a terminal that cannot sync the reference symbols (the tester loads
|
|
//--- auxiliary symbols from the terminal, not the server) must still get its diagnostics, with
|
|
//--- the gap stated rather than hidden.
|
|
if(m_ensembleMember && g_ensembleChartMiReportDone)
|
|
{
|
|
//--- see g_ensembleChartMiReportDone at the top of this file
|
|
m_miReportDone = true;
|
|
//--- Same reasoning as the tuner's adopt branch above: published, not just printed, so the row
|
|
//--- says why it is not repeating the measurement.
|
|
PublishStatus(ID + " : reusing the chart's information report...");
|
|
Print(ID + ": MI diagnostics already measured by another ensemble member on this chart - "
|
|
"same features, same labels, same answer. Skipped (saves the slowest part of the "
|
|
"ensemble's warm-up; the first member's report above is this model's too).");
|
|
//--- ...BUT THE GEOMETRY IS NOT A REPORT, IT IS A DECISION, and skipping the chain that makes it
|
|
//--- is not the same as declining it. "Same features, same labels, same answer" was true of the
|
|
//--- MI numbers and FALSE of the barrier the donor adopted: it relabelled itself and left the
|
|
//--- other three on their own derived pair, so the ensemble trained on two different targets and
|
|
//--- the joint gate certified the average as if it were one. Take the donor's pairing here, on
|
|
//--- the same reasoning the indicator settings are taken above - members training on divergent
|
|
//--- targets are not an ensemble.
|
|
//--- Guarded on era 0 for the same reason the donor's own adoption is: relabelling a partly
|
|
//--- trained net moves the target out from under weights already fitted to the old one.
|
|
if(g_ensembleChartGeomAdopted && m_eraCount == 0 && g_ensembleChartGeomSl > 0.0
|
|
&& g_ensembleChartGeomTp > 0.0
|
|
&& (m_derivedSlMult != g_ensembleChartGeomSl || m_derivedTpMult != g_ensembleChartGeomTp))
|
|
{
|
|
PrintFormat("%s: adopting the barrier geometry the chart's scan chose - %.2f*ATR / %.2f*ATR."
|
|
" This member never ran the scan (the MI chain runs once per chart), and keeping"
|
|
" its own derived pair would put this ensemble's members on DIFFERENT targets"
|
|
" while the orchestrator averages their votes as one.",
|
|
ID, g_ensembleChartGeomSl, g_ensembleChartGeomTp);
|
|
ApplyAdoptedGeometry(g_ensembleChartGeomSl, g_ensembleChartGeomTp,
|
|
g_ensembleChartGeomSlMode, g_ensembleChartGeomTpMode);
|
|
}
|
|
}
|
|
else
|
|
if(m_crossAsset.IsReady() || m_miReportDeferrals >= MI_REPORT_MAX_DEFERRALS)
|
|
{
|
|
//--- THE LONGEST SINGLE STRETCH OF THE WARM-UP - the MI suite, the lag profile, the excursion
|
|
//--- targets and the geometry scan, each with its own few-hundred-draw permutation null - and
|
|
//--- until now it published NOTHING. The tuner announced itself and then handed over to a
|
|
//--- chain that runs many times longer in silence, so the panel's last word during most of
|
|
//--- the warm-up described a step that had already finished.
|
|
PublishStatus(ID + (m_ensembleMember
|
|
? " : measuring feature/label information for the whole chart..."
|
|
: " : measuring feature/label information..."));
|
|
ReportFeatureLabelInformation();
|
|
if(m_ensembleMember && m_miReportDone)
|
|
g_ensembleChartMiReportDone = true;
|
|
}
|
|
else
|
|
m_miReportDeferrals++;
|
|
}
|
|
Train(StartTrainBar);
|
|
}
|
|
#endif // WARRIOR_AIBASE_AUTOTUNE_MQH
|